AI-Written Firmware: Grading AI Tools With 220 Tests

The numbers I cited in the last two posts come from one place. The 35%p gap, the DMA, ISR failure patterns — all of it. I built an LLM benchmark for embedded firmware, called it EmbedEval, and ran it. Today I’m releasing the benchmark itself.

Repo: github.com/Ecro/embedeval. Apache-2.0.

Why another benchmark

There’s no shortage of LLM coding benchmarks. HumanEval does general Python functions, SWE-bench does repo-level Python issues, Aider Polyglot does multi-language edits. All good. None of them go near embedded firmware. On the embedded side there’s EmbedAgent (ICSE’26), covering Arduino, MicroPython, Pico, and ESP-IDF (ESP32). The ESP-IDF 29.4% I quoted in post 1 came from there. But most of production embedded is just missing: Zephyr, STM32 HAL, FreeRTOS, Linux kernel drivers, Yocto. Still a blind spot.

I wanted to measure exactly one thing. When you hand an LLM some embedded code to write, how badly does it fall apart if the prompt never spells out the safety patterns? I wanted the gap I’d seen in post 1 as a number.

What it measures

233 cases. 185 public, plus 48 held-out that live in a separate repo so they stay out of training data. 23 categories, 6 platforms. Here’s the pipeline each case goes through.

L0 through L3 are the boring layers. L4 is the one that matters. The complaint that L3 is just regex heuristics is fair, so I bolted on L4 mutation testing. I plant bugs in the reference code on purpose: strip a volatile, flip a lock order, drop a header. Then I check whether L0/L3 actually catch them. If they don’t, the check is what’s broken, and it says nothing flattering about the model. So L4 measures one thing: how much you can trust the benchmark itself, not any single model’s score.

Default mode skips Docker and runs L0 plus L3 only. It’s the fastest path, and it separates models well enough on its own. L1/L2 kick in only when you’ve got a Docker SDK build environment.

n=3 results

Sonnet 4.6 and Haiku 4.5, three runs apiece. I computed the Wilson 95% CI over 699 pooled trials (233 cases × 3 runs).

Model	pass@1 (n=3 mean)	95% CI	Case stability
Sonnet 4.6	68.0%	[64.4%, 71.3%]	87%
Haiku 4.5	56.9%	[53.2%, 60.6%]	73%

The CIs don’t overlap. That 11.1%p gap between the two is statistically significant (p < 0.05). Case stability is the fraction of cases where all 3 runs agreed, all pass or all fail. For Sonnet, 143 of 233 pass every time, 60 fail every time, 30 are flaky. Haiku: 99 / 71 / 63. At n=3 the stable/flaky line is noisy, so v0.2 bumps it to n=5.

Where it’s weak (DMA, ISR, Threading)

Break pass@1 out by category and the pattern jumps out.

The declarative stuff holds up on both models. device tree, Kconfig, boot sequence all land between 90 and 100%. What collapses is consistent. The numbers below are pooled n=3. Post 2 quoted the early n=1 runs, so the per-category absolutes read differently, and a few categories like ISR even flip the model ranking. At 9 TCs × n=3, a flip that size is still inside the noise band.

• DMA cache coherency: Sonnet 31% / Haiku 8%. Exactly the pattern from post 2.
• ISR concurrency: Sonnet 23% / Haiku 38%. volatile, memory barriers, shared state across ISR and thread.
• Threading: Both 33%. Mutex ordering, deadline handling.
• Memory optimization: Sonnet 67% / Haiku 33%. memory domain, slab allocator.
• Storage: Sonnet 54% / Haiku 31%. Flash wear leveling, NVS lifecycle.

These five areas are the implicit knowledge gap from post 1, made concrete. They lean on context you can’t see inside the function body: datasheets, interrupt context, cache behavior, flash wear. They overlap almost perfectly with the 6 patterns from post 2.

Limitations (first)

Things I want to flag before anyone else gets to.

1. Only two models. Only Sonnet 4.6 and Haiku 4.5 got the n=3 treatment. GPT, Gemini, DeepSeek, Qwen are v0.2 work. v0.1 went for statistical confidence over breadth: run the same model enough times to pin down the CIs. The contributing workflow is wired up so new model results can land via PR.

2. Heavy Zephyr bias. Roughly 81% of the 233 are Zephyr cases. ESP-IDF, STM32 HAL, FreeRTOS, Linux driver, and Yocto sit at 5 to 10 cases apiece. v0.2 plans to grow FreeRTOS and Linux driver coverage. The reason is dull: native_sim and Docker builds behave best on Zephyr, so more cases can reach L1/L2 verification.

3. L1/L2 only on half. Only about 106 of the 233 cases go through L1 compile and L2 runtime. The rest target hardware boards (nrf52840dk and friends) or ESP-IDF/STM32, which native_sim can’t run. Those get L0+L3 only. METHODOLOGY spells out which is which, case by case.

4. Single-file scope. One case, one file. Multi-file scaffolding (header/source split, CMakeLists, Kconfig consistency) waits for v0.3. It’s a big slice of what LLMs do day-to-day, but the verification pipeline needs a rebuild for it.

5. L3 is regex heuristics. L4 mutation testing keeps it honest, but the ceiling is real. Actual semantic analysis only shows up when L1+L2 run, and that’s half the cases.

Benchmark roadmap

ROADMAP lays out v0.2 / v0.3 / v1.0. The order: expand model coverage, add FreeRTOS cases, HF Space submission flow, multi-file scaffolding, then agent mode work.

Right now, the three PRs that would help most:

• New model results: run n=3 with your API key, PR results/runs/<date>_<model>/
• Cases in weak categories: if you’ve seen production failure patterns in DMA, ISR, threading, contribute them as cases
• Stronger L3 checks: if you’ve hit false positives/negatives in a category, add to negatives.py

Three issue templates are ready: model evaluation request, case contribution, methodology question. Criticism is welcome. A problem found after release costs more than one caught before it.

• Repo: github.com/Ecro/embedeval
• Interactive leaderboard: huggingface.co/spaces/ecro/embedeval
• Methodology: docs/METHODOLOGY.md
• Roadmap: ROADMAP.md

If you ever wondered where the numbers cited across this series came from, you can go run them yourself.

---

The next post is about moving these numbers. Risk tiers (Critical / High / Moderate / Low) pulled from the category heatmap, a per-model selection guide for Sonnet vs Haiku, 9 module-type prompt templates (ISR, DMA, sensor, OTA…), and 9 one-line grep checks you drop into pre-commit. From measurement to practice.