Five boards are in fab. The next real number comes off a bench, not a build.
On the bench
SpotOnin fab
An nRF54L15 sensor node on Zephyr, sampling one ICM-42688-P at 1kHz and pulling modal features out of the impact window on-device.
Firmware and app are code-complete: FLASH 253,464 B, RAM 159,589 B, held to a baseline by a CI gate that fails on 2% drift.
The boards went on order August 2 and the order was cancelled the same day. I had specced bare boards to solder myself, then looked properly at U2: a 14-pad LGA on 0.5mm pitch with the terminals underneath it. Nothing on my desk solders that. Re-ordered through the assembly service, which split it into two orders because a board with parts on it cannot ship next to a 3D-printed part.
Five boards, 4-layer FR-4 at 0.8mm, 21mm across, ENIG. Two parts stay unplaced on purpose and go on by hand once they arrive. Paid, in production, nothing left on my side.
What arrives decides it. Fit check, then a destructive bench for insertion load and axial retention, and past that is the first kill gate. If the signal does not survive the real form factor, it stops there.
R4 EVT · 0.8mm 4L · DRC 0Industrial designBattery door · CR1632Assembly · 24mm across
v0.49.1. The thing that reads a repo and builds that repo its own agent harness. 48,937 lines, 5,997 tests, 180 SPECs.
The work right now is subtraction. Opus 5 handles in one call what used to need a scripted sequence, so the harness is on a diet: four stages had their fixed call sequences collapsed into single calls, Pass 1.5 is cut, the worktree config axis collapsed to one key, and three audit commands were retired. Every stage removed is context the model no longer has to carry.
A stronger model does not call for a more complicated harness. What matters now is how much of it you can delete.
v0.2.0. 267 firmware cases across Zephyr, ESP-IDF, STM32 and Yocto, scored through five layers that end in a real cross-compile.
Sonnet 5 scored 67.0% pass@1 against Sonnet 4.6’s 68.0%. On the 232 cases both models saw, that is 0.9 points down, which is a tie. A model generation went by with no measurable gain on embedded firmware.
Every row on that board is still an Anthropic model, which makes it a report rather than a leaderboard. Open-weight models and GPT are going through the harness now. No numbers until the runs finish.
pass@1 by category · n=3 mean · all three models break in the same places leaderboard →
Chewing on
Whether an LLM-driven improvement loop can actually make a firmware codebase more honest. Three times this month I chased a problem down and the answer came back that the check was wrong, not the code. Getting an agent to keep fixing code may be the easy half; getting it to judge what it must not touch looks like the harder one.
Open to
Conversations about putting AI agents on real embedded hardware. What breaks on the way from a demo to a product, firmware code audits, reliability, verification, and getting agents to behave on a hardware codebase.
