AI-Written Firmware: Measuring the Gap With 233 Tests

The numbers I cited in the last two posts all come from one place. The 35%p gap, the DMA, ISR failure patterns: every one of them. I built an LLM benchmark for embedded firmware, called it EmbedEval, and ran it myself. Today I’m releasing the whole benchmark.

The repo is github.com/Ecro/embedeval, under Apache-2.0.

Why another benchmark

LLM coding benchmarks are not in short supply. HumanEval covers general Python functions, SWE-bench covers repo-level Python issues, and Aider Polyglot covers multi-language edits. Each of them is useful. None of them goes near embedded firmware. On the embedded side there is EmbedAgent (ICSE’26), which covers Arduino, MicroPython, Pico, and ESP-IDF (ESP32). The ESP-IDF 29.4% I quoted in post 1 came from there. But most of production embedded is simply absent: Zephyr, STM32 HAL, FreeRTOS, Linux kernel drivers, Yocto. Nobody is measuring that part yet.

What I wanted to measure was a single thing. When you hand an LLM some embedded code to write, how badly does it fall apart if the prompt never spells out the safety patterns? I had seen that gap with my own eyes in post 1, and I wanted it written down as a number.

What it measures

There are 233 cases. 185 are public, and 48 are held out in a separate repo so they stay out of training data. They span 23 categories and 6 platforms. Each case goes through the pipeline below.

L0 through L3 are the layers you would expect. L4 is the one I care about most. The complaint that L3 is only regex heuristics is fair, and that is why I added L4 mutation testing. I plant bugs in the reference code on purpose: strip a volatile, flip a lock order, drop a header. Then I check whether L0/L3 actually catch them. If they don’t, the model has not done anything clever. The check is what’s broken. So what L4 measures, in the end, is how far you can trust the benchmark itself.

The default mode skips Docker and runs only L0 and L3. It is the fastest path, and it separates the models well enough by itself. L1/L2 run only when a Docker SDK build environment is installed.

n=3 results

I ran Sonnet 4.6 and Haiku 4.5 three times each. The Wilson 95% CI is computed over 699 pooled trials (233 cases × 3 runs).

Model	pass@1 (n=3 mean)	95% CI	Case stability
Sonnet 4.6	68.0%	[64.4%, 71.3%]	87%
Haiku 4.5	56.9%	[53.2%, 60.6%]	73%

The two CIs don’t overlap. The 11.1%p gap between the models is statistically significant (p < 0.05). I defined case stability as the fraction of cases where all 3 runs agreed, either all pass or all fail. For Sonnet, 143 of 233 passed every time, 60 failed every time, and only 30 were flaky. Haiku came out at 99 / 71 / 63. At n=3 the line between stable and flaky is bound to be noisy, so v0.2 raises it to n=5.

Where it’s weak (DMA, ISR, Threading)

When I break pass@1 out by category, the pattern is easy to see.

Both models handle the declarative areas well. device tree, Kconfig, and boot sequence all land between 90 and 100%. The areas that collapse are always the same ones. The numbers below are pooled n=3. Post 2 quoted the early n=1 runs, so the per-category absolutes read differently from what I wrote then, and a few categories like ISR even flip the model ranking. At 9 TCs × n=3, a flip that size still sits inside the noise.

• DMA cache coherency: Sonnet 31% / Haiku 8%. The same pattern I looked at closely in post 2.
• ISR concurrency: Sonnet 23% / Haiku 38%. volatile, memory barriers, shared state across ISR and thread.
• Threading: Both 33%. Mutex ordering, deadline handling.
• Memory optimization: Sonnet 67% / Haiku 33%. memory domain, slab allocator.
• Storage: Sonnet 54% / Haiku 31%. Flash wear leveling, NVS lifecycle.

These five areas are what the implicit knowledge gap from post 1 looks like in practice. Each depends on context you can’t see from inside the function body: datasheets, interrupt context, cache behavior, flash wear. They overlap almost exactly with the 6 patterns from post 2.

Limitations (first)

Before anything else, there are a few things I want to put on the table myself.

To start with, there are only two models. Only Sonnet 4.6 and Haiku 4.5 got the n=3 runs; GPT, Gemini, DeepSeek, and Qwen are v0.2 work. For v0.1 I chose statistical confidence over breadth, running the same model enough times to pin down the CIs instead of adding more models. The contributing workflow is in place so new model results can arrive via PR.

The set also leans hard toward Zephyr. Roughly 81% of the 233 are Zephyr cases, while ESP-IDF, STM32 HAL, FreeRTOS, Linux driver, and Yocto have 5 to 10 cases each. The reason is plain: native_sim and Docker builds work best on Zephyr, so more of its cases can reach L1/L2 verification. v0.2 plans to grow FreeRTOS and Linux driver coverage.

Even L1/L2 runs on only half. Only about 106 of the 233 cases go through L1 compile and L2 runtime. The rest target hardware boards (nrf52840dk and friends) or ESP-IDF/STM32, which native_sim can’t run, so those are measured with L0+L3 only. METHODOLOGY records which is which, case by case.

The scope is a single file, too. One case is one file, so multi-file scaffolding (header/source split, CMakeLists, Kconfig consistency) is not measured and waits for v0.3. It is a large part of what LLMs do every day, but measuring it means rebuilding the verification pipeline from the start.

And L3 is, in the end, regex heuristics. L4 mutation testing keeps it honest, but the ceiling is real. Real semantic analysis only happens when L1+L2 run, and that is half the cases.

Benchmark roadmap

ROADMAP lays out v0.2 / v0.3 / v1.0. In order: expand model coverage, add FreeRTOS cases, an HF Space submission flow, multi-file scaffolding, then agent mode work.

Right now, three kinds of PR would help most. If you have an API key, run a new model at n=3 and PR the results to results/runs/<date>_<model>/. If you have seen production failure patterns in the weak categories like DMA, ISR, threading, contribute them as cases. And if you have hit false positives/negatives from an L3 check in some category, strengthen it by adding to negatives.py.

There are three issue templates: model evaluation request, case contribution, methodology question. I welcome criticism. A problem found after release costs more than one caught before it.

• Repo: github.com/Ecro/embedeval
• Interactive leaderboard: huggingface.co/spaces/ecro/embedeval
• Methodology: docs/METHODOLOGY.md
• Roadmap: ROADMAP.md

If you ever wondered where the numbers in this series came from, you can now run them yourself.

---

The next post is about moving these numbers. It covers risk tiers (Critical / High / Moderate / Low) drawn from the category heatmap, a per-model selection guide for Sonnet vs Haiku, 9 module-type prompt templates (ISR, DMA, sensor, OTA…), and 9 one-line grep checks you can put in pre-commit. It is where measurement turns into practice.