Wow, I didn’t think it would take me three *years* to write another blog post, but, well, here we are. Post pandemic, post job change, and post child-in-school. Funny how many changes can happen in so few years!

I’ve been working on a side project with a client for awhile now which required going more in depth with infrared (IR) temperature sensors. Initially we just used multiple MLX90614 sensors to measure temperatures at several “points”, but eventually we transitioned the project to using a full thermal camera (MLX90640). We learned a lot, but eventually the client decided to go in a different direction for the project. While I won’t be working on it anymore, I wanted to blog about some of the tech, since there were some really cool, practical problems we had to solve. Namely:

• How to visualize a (possibly) small temperature change over small amount of time?
• How to record and playback the same visualization
• How to deal with noise-y IR sensors?
• How to transfer a large amount of data to another device?
• How to take more readings to make the visualization more responsive?
• How to deal with connectivity issues over larger distances?
• How to deal with powering the device?

I think I could write a blog post about each of these topics, to be honest, but I’ll keep this one focused on two problems: how to visualize a temperature change, and how to transfer a large amount of data to another device.

Sensor and Microcontroller

We tried a few different iterations of the sensor capture device in order to get the quality of readings from it that we wanted to get. The main problem was the client wanted to get multiple readings a second, but the basic microcontrollers we had couldn’t achieve it. We ended up using a Raspberry Pi Zero 2 W (a difficult feat during the chip shortage, let me tell you!), but for this blog post, I’ll use a earlier version of the final device. The basic circuit: an ESP32-S2 feather, the MLX-90640, and a few connecting wires. The thermal camera just uses I2C, so it’s easy to connect by connecting SCL and SDA on the feather:

Just as we tried different microcontrollers, we also tried different programming languages to improve the performance of the device. In the end, we ended up using python on the Raspberry Pi, but with the feather the easiest code is just using basic Arduino:

#include <Adafruit_MLX90640.h>
#include <Wire.h>
#include <PubSubClient.h>
#include <WiFi.h>
#include "arduino_secrets.h"

Adafruit_MLX90640 mlx;
#define ROWS 24
#define COLUMNS 32
float frame[ROWS * COLUMNS];

#define UPDATE_INTERVAL 500

char ssid[] = SECRET_SSID;
char pass[] = SECRET_PASS;

WiFiClient wifiClient;
PubSubClient mqttClient(wifiClient);

const char broker[] = "192.168.68.121";
int port = 1883;
const char* topic = "helloworld";
const char* clientname = "arduino";

void setup() {
  Wire.begin();
  Wire.setClock(800000);
  Serial.begin(9600);
  delay(3000);    // wait for serial monitor

  Serial.print("-- Attempting to connect to WPA SSID: ");
  Serial.println(ssid);
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    delay(1000);
  }
  Serial.println("");
  Serial.print("Connected! IP address: ");
  Serial.println(WiFi.localIP());
  Serial.println("-- You're connected to the network");

  Serial.println();
  Serial.print("--- Attempting to connect to the MQTT broker: ");
  Serial.println(broker);
  mqttClient.setServer(broker, port);
  mqttClient.setBufferSize(100000);

  if (!mqttClient.connect(clientname)) {
    Serial.print("MQTT connection failed!  State: ");
    Serial.println(mqttClient.state());
    while (1);
  }
  Serial.println("--- You're connected to the MQTT broker!");
  Serial.println();

  Serial.println("---- Starting thermal camera MLX90640");
  if (! mlx.begin(MLX90640_I2CADDR_DEFAULT, &Wire)) {
    Serial.println("MLX90640 not found!");
    while (1) delay(10);
  }
  Serial.print("Found MLX90640! Serial number: ");  
  Serial.print(mlx.serialNumber[0], HEX);
  Serial.print(mlx.serialNumber[1], HEX);
  Serial.println(mlx.serialNumber[2], HEX);
  mlx.setMode(MLX90640_CHESS);
  Serial.print("Current mode: ");
  if (mlx.getMode() == MLX90640_CHESS) {
    Serial.println("Chess");
  } else {
    Serial.println("Interleave");    
  }
  mlx.setResolution(MLX90640_ADC_18BIT);
  Serial.print("Current resolution: ");
  mlx90640_resolution_t res = mlx.getResolution();
  switch (res) {
    case MLX90640_ADC_16BIT: Serial.println("16 bit"); break;
    case MLX90640_ADC_17BIT: Serial.println("17 bit"); break;
    case MLX90640_ADC_18BIT: Serial.println("18 bit"); break;
    case MLX90640_ADC_19BIT: Serial.println("19 bit"); break;
  }
  mlx.setRefreshRate(MLX90640_16_HZ);
  Serial.print("Current frame rate: ");
  mlx90640_refreshrate_t rate = mlx.getRefreshRate();
  switch (rate) {
    case MLX90640_0_5_HZ: Serial.println("0.5 Hz"); break;
    case MLX90640_1_HZ: Serial.println("1 Hz"); break; 
    case MLX90640_2_HZ: Serial.println("2 Hz"); break;
    case MLX90640_4_HZ: Serial.println("4 Hz"); break;
    case MLX90640_8_HZ: Serial.println("8 Hz"); break;
    case MLX90640_16_HZ: Serial.println("16 Hz"); break;
    case MLX90640_32_HZ: Serial.println("32 Hz"); break;
    case MLX90640_64_HZ: Serial.println("64 Hz"); break;
  }

  Serial.println("---- Finished thermal camera MLX90640");
}

void loop() {
  String data = generateReadingsJSON(millis());
  int str_len = data.length() + 1;
  char char_array[str_len];
  data.toCharArray(char_array, str_len);
  mqttClient.publish(topic, char_array);
  delay(UPDATE_INTERVAL);
}

String generateReadingsJSON(long timestamp) {
  char data[4000];
  String values = generateThermalCameraCellJSON();  
  snprintf(data, sizeof(data), "{\"t\": \"%lu\", \"s\": [{\"n\": \"thermal camera\", \"t\": \"tmatrix\", \"r\": \"%s\", \"c\": \"%s\", \"v\": \"", timestamp, String(ROWS), String(COLUMNS));
  String string = String(data);
  string += values;
  string += "\"}]}";
  return string;
}
String generateThermalCameraCellJSON() {
  if (mlx.getFrame(frame) != 0) {
    Serial.println("Failed to read from thermal camera");
    return "";
  }
  String json = "";
  for(uint8_t row = 0; row < ROWS; row++) {
    for(uint8_t column = 0; column < COLUMNS; column++) {
      float temperature = frame[row * COLUMNS + column];
      char data[10];
      snprintf(data, sizeof(data), "%.2f", temperature);      
      json += String(data);
      if(!((row == ROWS - 1) && (column == COLUMNS -1))) {
        json += ",";
      }
    }
  }
  return json;
}

There are a few gotchas here to point out: I had to use the PubSubClient Arduino library because it let me set the buffer size to allow for a larger buffer size (I set it to 100000 as a test and it worked, so I left it, haha). The second is JSON is (not surprisingly) difficult to generate on Arduino. I tried the Arduino JSON library but in the end found it simpler to generate a simple string using snprintf.

Note that the MLX90640 has 24 rows and 32 columns, so you can take a temperature reading at every cell in that array, for a total of 768 individual temperatures. While that doesn’t seem like a lot, remember that we wanted to take several readings a second. For that reason, the JSON message is purposefully short-handed to make it as small as possible. I wanted to capture all 768 temperatures at the same point in time, so I created a long string of all the temperature values, and also included the number of rows and columns so the receiving server could parse the values back into a meaningful array again. Here’s an annotated version of the JSON string the microcontroller is sending:

{
	"t": "985606",  // timestamp
    "s": [			// sensor
        {
			"n": "thermal camera",	// name
			"t": "tmatrix",			// type
			"r": "24",				// rows
            "c": "32",				// columns
            "v": "29.37,27.97,28.34,28.26,28.60,28.28,27.96,28.73,28...."	// values
        }
    ]
}

Server

Once we got the device reading temperatures, we had to get the values back to a server so we could analyze and display them in a meaningful way back to the viewer. For the final project, I wrote a basic Java REST service to accept POSTs from the microcontroller and serve GETs to display the results. However, for this blog post I thought I’d try to keep things a bit simpler and use MQTT. Simplistically, MQTT is just a protocol that supports publish and subscribe. The microcontroller publishes messages TO an MQTT broker (so all it does is read and send, read and send, read and send). Then an application subscribes to messages FROM the broker (so all it does is read and display, read and display, read and display). Architecturally, I liked the design with MQTT better because the broker makes a clean division between the sender and the receiver. The PubSubClient Arduino library handled MQTT easily (connect and publish functions are built in).

For a MQTT broker, I used Apache Artemis (formerly known as Apache ActiveMQ). I used the base installation, and the hardest part that I had to figure out was how to setup Windows Firewall rules to allow TCP on the local network’s port 1883. Most of the configuration just worked right out of the box, though I wasn’t concerned for this weekend hack about things like user names, passwords, and other important security considerations (which, of course, for a real project you should worry about, haha). The nicest feature of Artemis is the local console that you can use to send test messages and identify the connections. It saved me a lot of debugging time!

Monitor

Finally, we had to figure out how to display the temperatures from the camera to a user to show any changes happening in real time. The easiest way we thought of was a basic heatmap, similar to what you’d see on a heat gun. For that, I used the library d3.js, though I used an older version of the library I was more familiar with. To pull the messages from the MQTT broker, I set up a websocket (supported out of the box by Artemis), some simple transformations via maps, and a bunch of the d3 libraries. The code is a bit long, so I recommend viewing it on github.

There are a few parts of the code I will point out. The first is setting up the websocket, which was surprisingly easy. I used the Paho client to connect, and just loaded it from a CDN:

function connectToMQTT() {
	console.log("Connecting to " + host + " " + port);
	mqtt = new Paho.MQTT.Client(host, port, "helloworldjsclient");
	var options = {
		timeout: 3,
		onSuccess: onConnect,
		onFailure: onConnectFailure
	}
	mqtt.connect(options);
	
}
function onConnect() {
	console.log("Connected to mqtt server");
	mqtt.subscribe("helloworld");
	mqtt.onMessageArrived = onMessageArrived
}
function onConnectFailure() {
	console.log("Failed to connect to mqtt server");
}

Yup, pretty straightforward. Once it connected to the broker, I setup the subscription to the helloworld topic (the same one the microcontroller is publishing to) and then did a quick transformation (mapValuestoReadings) to turn the message JSON (see above) to an array of objects, which was easier to use with d3. Once the data was ready, I sent it to the method renderTMatrix to actually display the heatmap.

function onMessageArrived(message) {
	try {
		json = JSON.parse(message.payloadString);
		for(var i = 0; i < json.s.length; i++) {
			var s = json.s[i];
			var sensor = {
				"name": s.n,
				"type": s.t
			}
			if(sensor.type == "tmatrix") {
				sensor.rows = s.r;
				sensor.columns = s.c;
				sensor.reading = mapValuesToReading(s.v, sensor);
				renderTMatrix(sensor);
			}
		}
	} catch(e) {
		console.log(e);
	};
}

Originally I had thought that the microcontroller device might have multiple sensors that we’d want to capture and display on the monitor, so that’s why the message payload had an array for sensors and the check for sensor type in the code above. You can ignore it for this example, but you can extend it to include multiple sensors if needed.

The first time the sensor is drawn, the graph squares, legend, tick marks, and axes have to be drawn. Then once the initial graph is drawn, subsequent updates simply just have to update the color of each of the squares. However, because we don’t know when we’ll get the second message for the heatmap (since messaging is asynchronous, the renderTMatrix method includes a check to not initialize the heatmap more than once. And yeah, there’s probably a nicer way to do this than using a (gasp) global variable, but hey, it worked:

function renderTMatrix(sensor) {
	if(initializing) {
		// do nothing -- ignore this message until we're fully initialized
	} else if(initialized) {
		updateHeatmap(sensor);
	} else {
		console.log("Intializing heatmap");
		initializing = true;
		initializeHeatmap(sensor);
		initialized = true;
		initializing = false;
		console.log("Done initializing heatmap");
	}
}

Here’s a picture of the final monitor (just an image of my hand waving hello). You’ll have to use your imagination, but the squares update about twice a second. It’s sort of choppy, but fast enough to at least see temperature changes occurring quickly in a small area. And not a whole lot of code to get the whole thing to work!

Final Thoughts

I’m going to miss this project. Most of the hobby circuits I’ve done haven’t had such…practical details to worry about. But it’s nice to be able to share some technical parts of the project. I’ve made the code available on github, and comments / suggestions are always welcome!

…but now I have to find another project haha!