What it is
A sensor reading starts on a microcontroller and ends up on a graph in someone's browser. Built during my year at Axisops.
Devices publish over MQTT. A RabbitMQ broker routes the traffic and holds it while a set of small Python services work through it. An internal API owns the database and is the only way to it; a separate public API serves users, their devices and the data itself. A React front end puts that in front of the person, who can graph it or pull it out as CSV.
A fleet of devices, one pipeline, firmware to browser.
What I built
Firmware. Across a range of chips, mostly Raspberry Pi Picos and ESP32s, reading everything from simple digital sensors up to driving motors. Written in MicroPython and C, with a little CircuitPython.
Messaging. Devices publish over MQTT, with RabbitMQ behind it bridging MQTT and AMQP. Several queues, with routing, retries and dead-letter queues for messages that could not be delivered. One central broker meant every API and helper service talked to the same place, it scaled as the number of devices grew, and it gave a route back to the microcontrollers rather than only away from them.
Storage. TimescaleDB, because every reading arrives with a timestamp and almost every question asked of the data is a time range. Timescale buckets rows by time, which is what makes the dashboard's graphs and CSV exports fast. Users, devices and readings are properly related, so the data stays correct as well as efficient.
Two APIs, split by who is calling. An internal one that owns the database and is the only thing permitted to touch it, so every other component asks it rather than running its own queries. A separate public one for everything outside: users, their devices, and the data itself. The split keeps the database behind a single door, and means the external surface can change shape without anything internal noticing.
Services between the two. A handful of small Python services sit behind the broker and reach storage only through the internal API, so the layering holds all the way down rather than being a rule the edges follow and the middle ignores.
Device lifecycle. A device is bound to a user's account, so its readings arrive as theirs, and can be released again later. That binding is what decides who is allowed to see a reading, and what happens to the history when a device changes hands.
Access control. Signed-in people carry bearer tokens; devices authenticate with their own keys rather than borrowing a user's session. Admin powers are a separate role rather than a flag on an ordinary account. Credentials are hashed and salted, never stored as given, and the identifiers used externally are opaque.
Tests. Both APIs have suites that run every endpoint as an ordinary user and again as an admin, so the checks cover who is refused and not just what succeeds.
Frontend. A React and TypeScript single-page app against the public API, with token-backed sessions and admin-only routes. It covers your devices and their readings, graphed and exportable as CSV, account settings, and admin screens for users, devices and device types.
Alongside the build I wrote the documentation and gave the technical demo.
What I learned
RabbitMQ is where I lost the most time, and the one I still remember was a binding that did not match. Messages were published successfully, routed nowhere, and consumed by nobody. Nothing failed: the publisher got its acknowledgement, no consumer raised anything, and the only symptom was data that should have arrived and had not. By the time I noticed the asynchronous problem, whatever caused it had finished and moved on.
Making one service the only thing that touches the database is the decision I would defend hardest. It felt like a pointless extra hop every time I added an endpoint. But the schema could change without hunting through every service that had read a table, and a rule had one place to live rather than several that all had to agree.
Each piece worked on its own long before the system did. The time went on getting them to keep talking when one of them failed, which turned out to be a different skill from making any of them work.
The tests that earned their keep were the ones checking who gets refused rather than who gets through.
What I would add next
Two things I had started looking at and did not finish:
- Device firmware updates delivered over MQTT, so the fleet could be updated remotely rather than by hand
- LoRaWAN, to reach devices well outside wifi range
Personal thoughts
I really enjoyed this project. It was the biggest thing I had built, and getting the whole pipeline working end to end was the best part. I'm grateful I got to work on it, alongside plenty of other things, during my year at Axisops.