AI-Written Firmware: What I Didn't Tell It
A junior wrote a logging daemon with an AI tool. It passed my review, ran fine, and later wore out an eMMC. The risk was in what the prompt left out.
On this page
What makes LLM-generated embedded code dangerous is not that it is wrong. It is that it goes wrong without anyone noticing. When I tell it exactly what to do, it usually delivers. But the moment the task needs domain knowledge nobody put into words, the pass rate falls off a cliff. The coding benchmarks most people quote don’t catch this gap. I learned it from one logging daemon that passed my review and went out to the field.
Code That Silently Kills Your eMMC
I was reviewing a junior engineer’s code on an i.MX8M Mini BSP project that day. It was a CAN logging daemon, and I could tell at a glance that an AI tool had written it. Over 200 lines, with signal handling, daemonization and PID file management, nothing missing, all of it production-grade structure. A junior rarely writes at that level of completeness alone. The core loop looked like this.
/* AI-generated CAN logging daemon (core loop) */
#define DEFAULT_LOGFILE "/var/log/can.log"
#define LOG_FLUSH_INTERVAL 10 /* seconds */
FILE *fp = fopen(logfile, "a"); /* logfile = "/var/log/can.log" */
alarm(LOG_FLUSH_INTERVAL); /* SIGALRM every 10s to flush */
while (running) {
struct can_frame frame;
ssize_t nbytes = read(sock, &frame, sizeof(frame));
if (nbytes < 0) {
if (errno == EINTR) {
if (flush_flag) { fflush(fp); flush_flag = 0; alarm(LOG_FLUSH_INTERVAL); }
continue;
}
break;
}
log_frame(fp, iface, &frame, &ts);
if ((frame_count + err_count) % 1000 == 0)
fflush(fp); /* flush every 1000 frames */
}
To be honest, it was well built. Instead of calling fflush() on every frame, it buffers on a 10-second interval plus every 1000 frames. The error handling is there, and a signal shuts it down cleanly. I hit approve and moved on.
There was one thing I did not see then. The path was /var/log/can.log. That file sits on eMMC.
On our system, CAN messages arrive every 100ms. With 1000-frame buffering, that comes to a flush roughly every 100 seconds. Add the 10-second alarm on top and the eMMC sees thousands of writes a day. Appending to one file keeps landing on the same block group, even with wear leveling in place. That is how the P/E cycles on MLC/TLC eMMC wear down, a little at a time. With bad luck, the eMMC flips to read-only, or takes the boot partition down with it. The eMMC on the device running this daemon did fail in the end.
Once I found the cause, the fix was simple. Write to tmpfs instead of /var/log/, and only copy down to eMMC on a schedule.
/* Fix: tmpfs buffer + periodic eMMC flush */
#define DEFAULT_LOGFILE "/tmp/can_buffer.log" /* tmpfs (RAM) */
#define EMMC_FLUSH_INTERVAL 3600 /* save to eMMC once per hour */
That takes thousands of daily writes down to 24. I also set up logrotate so old files get cleaned. Missing it in review cost us. We only found out after the devices were deployed and the eMMC had failed.
Looking back, the file I/O itself had nothing wrong with it. Signal handling, buffering and error handling were all good enough to ship. Only one thing was missing: the fact that this code runs on eMMC. On a server the code would have been perfect. On an embedded device it wears out the hardware. Nobody told the AI tool to “think about eMMC wear,” so it had no reason to.
What stayed with me longer was that neither the junior nor I, the reviewer, knew this. The junior assumed clean AI code was right, and I approved it because it looked clean. I was the one in a position to catch it, I did not, and the code shipped to the field as it was.
35 Percentage Points: The Gap Benchmarks Hide
I wondered whether this was a problem with one logging daemon or something larger. So I built about 200 embedded test cases across 20 categories and ran each one in two versions.
- Explicit: “Declare a volatile int flag and set it in the ISR” — most pass
- Implicit: “Communicate between the ISR and main thread via a flag” (no mention of volatile) — pass rate drops noticeably
In my tests the gap came out around 35 percentage points. Verification was regex-based heuristics rather than a real runtime run, though, and the test cases lean heavily toward Zephyr. So I don’t put much weight on the exact number. What I saw was a shape more than a number: the gap between explicit and implicit prompts stays wide and consistent.
Those tests later grew into EmbedEval, a 233-case benchmark with a real compile and runtime layer. There, Sonnet 4.6 scored 68.0% and Haiku 4.5 scored 56.9%. The place where both of them fell apart was still the implicit-knowledge categories.
Coding benchmarks like HumanEval and SWE-bench are almost all explicit style. They say “implement this function” and spell out what they mean. But the hard part of embedded work lives in what you don’t write in the prompt: volatile, eMMC wear, cleanup ordering, timing margins. Almost nothing measures that.
So whenever I see “the LLM scores 90%,” I find myself asking: is that 90% with the answer already sitting in the prompt, or 90% where it had to bring the domain knowledge on its own?
The Same Gap Shows Up in Papers
The gap isn’t only in my tests. In EmbedAgent (2025, arXiv preprint), LLM pass@1 is 73.8% on MicroPython (Raspberry Pi Pico) and drops to 29.4% on ESP-IDF. MicroPython APIs like pin.value() show everything on the surface. ESP-IDF doesn’t: esp_err_t checks, the component system and menuconfig conventions are rules you are expected to follow without being told. It is the same implicit knowledge problem.
The IoT-SkillsBench (2026) result stings a little more. Injecting LLM-generated “helper context” into the prompt made Zephyr performance worse. The LLM had hardened its own wrong assumptions. Context written by a human expert, on the other hand, pushed the score up to 39/42. Not all context is equal. Dumping in any old context hurt, and only context drawn from real failure experience helped.
What I Changed
After that I changed only one thing. I put a silent failure checklist in the project config file (CLAUDE.md).
## Silent Failure Checklist
- eMMC writes -- high-frequency writes must use tmpfs buffering. No direct fflush
- Error paths -- LLM almost never does reverse-order cleanup on init failure
- IRQ context -- LLM inserts blocking calls (mutex, sleep) in interrupt handlers
- Device Tree -- pinctrl, clock required properties get omitted
- Timing margins -- LLM sets timeout = period with zero margin
Every time I catch one in review, a line gets added to the list. In three months with this checklist, over dozens of generated modules, the same failure types haven’t come back. Five lines in a file did better than dumping the whole datasheet into RAG. The LLM can’t pick “watch the eMMC write endurance” out of a 512-page reference manual. But it reads a one-line checklist entry just fine.
Is it a complete fix? It isn’t. I can only write down the traps I know about. The ones I don’t know are still hiding inside that gap. Still, as the traps pile up one by one, CLAUDE.md turns into a document where knowledge that used to go unsaid has been written down. The more experience the engineer has, the thicker the file gets, and the wider the area I can safely hand to an LLM.
So now, when I look at LLM-written embedded code, the first thing I ask is not “does it compile?” It is “what did I not tell it?” If no answer comes to mind for a given chunk of code, that code isn’t ready to hand to an LLM yet.
For a concrete way to solve this with context engineering, see Context Engineering.
Comments
Loading comments...