AI-Written Firmware: What to Hand Over, Where to Stop It
3-Tier Trust sorts PRs, then verification gates (spec, TDD, RAG, HIL, deterministic analyzer, more) take them to merge. Each catches its own defect class.
On this page
Right after the Part 3 benchmark post went up, someone asked: “So what do we actually do with this?” There is no single answer. It depends on the category. Run the EmbedEval n=3 data and LLM output falls into three buckets: safe to approve on its own, review required, and write it by hand from the start. What I find interesting is that the boundary doesn’t move however many tokens you spend. Datasheets and board identity sit outside the model’s context to begin with.
3-Tier Trust
| Tier | Categories | Sonnet | Haiku | PR Rule |
|---|---|---|---|---|
| 1. Auto-merge | device-tree, pwm, boot, kconfig | 90%+ | 90%+ | syntax review, then merge |
| 2. Expert review required | gpio, linux-driver, yocto, ota, power-mgmt, spi-i2c, ble | 67 to 90% | 45 to 83% | verify error paths + side effects |
| 3. Expert writes from scratch | DMA, ISR, threading, storage | below 55% | below 55% | LLM draft is reference only |
The numbers come from Part 3: the n=3 measurements over 233 TCs. I kept the threshold simple. Both models under 60%, with n=3 stability under 80%, means Tier 3. Why 60% and 80%? That is where the inequality “time to write from scratch ≤ time to polish an LLM draft into something that passes” stopped holding. It is a value I expect to re-measure when the next model generation ships.
The CAN logging daemon that killed an eMMC back in Part 1 belongs to Tier 3 storage (flash wear leveling, NVS lifecycle). The 6 failure patterns I pulled out in Part 2 land the same way: ISR (volatile), DMA cache coherency, error-path cleanup, nearly all of them gathered around Tier 3. Part 3 fixed the boundary in numbers. In this part I try to turn those numbers into a rule a team can run on.
Why More Tokens Don’t Move the Boundary
For ordinary software, quality follows a log curve as you spend more tokens. In embedded, the curve flattens at some point. The places where it flattens share something: the datasheet, the reference manual, real-time timing, and “which board am I sitting on right now.” All of them are missing from the context.
In EmbedAgent (ICSE 2026), datasheet RAG raised ESP-IDF from 29.4% to 65.1%. That’s 35.7 points recovered. What I read in that number is that context engineering raises the ceiling. It does not remove the ceiling. Even with whole datasheet pages pushed in through RAG, facts like “my board is mask set X of the i.MX8M Mini, and SDMA channel N is wired to PHY Y” still don’t cross over on their own.
Tier 3 Is Red, Not Grey
DMA category: Sonnet 31%, Haiku 8% (Part 3 n=3). One concrete case shows where those numbers come from.
Allocate an i.MX8M Mini SDMA buffer with kmalloc() instead of Linux’s dma_alloc_coherent API, and the SDMA can’t see the data held in the A53 L1 cache. The i.MX8M Mini reference manual states it plainly: SDMA sits outside the Cortex-A53 cache coherency domain. The fix takes one line.
/* SDMA on i.MX8M Mini is outside the A53 cache coherency domain.
* Use coherent allocation instead of kmalloc.
* dev = &pdev->dev (struct device *) from platform_driver probe.
* dma_handle: dma_addr_t, dma_buf: void *. */
dma_buf = dma_alloc_coherent(dev, DMA_BUF_SIZE, &dma_handle, GFP_KERNEL);
There are two reasons the LLM can’t write that line. The relevant page of the reference manual is not in its context, and the fact that “I’m on an i.MX8M Mini” is outside the prompt in the first place. You can iterate all day and it keeps not knowing the same thing. The review side has the same problem. To see kmalloc where dma_alloc_coherent belongs and call it broken, the reviewer already has to know that non-coherent SDMA exists. A reviewer who doesn’t know the answer turns the review into a rubber stamp. The eMMC death in Part 1 had exactly this shape: a junior merged an LLM-written logging daemon, and nobody in the review chain thought to suspect write amplification.
How to Stop It
The first thing I do is block auto-merge with CODEOWNERS. List the Tier 3 directories in .github/CODEOWNERS or a branch protection rule.
# Tier 3: expert review mandatory, no LLM auto-merge
drivers/dma/ @firmware-leads
kernel/threading/ @firmware-leads
drivers/storage/ @firmware-leads
Sonnet goes on Tier 2 only. Sonnet at 68% versus Haiku at 57%: that 11-point gap only pays for itself at Tier 2. Tier 1 runs fine on Haiku, and Tier 3 gets written from scratch by a person whichever model you attach. So if you route Sonnet calls to Tier 2 PRs only, the bill drops noticeably. The exact savings depend on your own team’s Tier mix (PR counts, token volume).
Tier 2 review looks only at error paths and side effects, because this is the stretch where an LLM halves the work time and exactly two spots are left uncovered. One is the error-path branches and reverse-order resource cleanup. The other is the side effects of recipe, DTS, and Kconfig changes. The second one caught our team once. We changed a single Yocto recipe, it touched the kernel DTB, and after boot the SPI flash write latency jumped from 50ms to 800ms. A review that only reads the recipe diff cannot catch that.
And the boundaries do not stay put. Run this for about six weeks and more than half the categories get reclassified. BLE GATT moved up from Tier 2 to Tier 3 once its timing sensitivity showed; i2c bring-up came down from Tier 3 to Tier 2 after we attached datasheet RAG. So this table is not something you set once and leave. It is an operating record you re-measure every quarter.
Verification Gates, in Three Layers
Those four moves are pieces of a larger pipeline. Code that makes it all the way through Tier 3 to a merge passes verification gates that each catch a different class of defect. Roughly, there are three layers: Implicit Context (inject facts at generation time), Review (static checks before merge), Test (runtime verification).
The first layer, Implicit Context, puts the facts into the LLM’s context at the moment of generation. One part of it is spec-driven generation: fix the requirements as a contract YAML first, so the LLM and the reviewer read the same contract. The other is datasheet RAG injection. Chunk the board reference manual, errata, and SDK examples, then feed them in as generation-time context. EmbedAgent ICSE 2026’s +35.7-point recovery is the quantitative data behind this layer.
The Review layer verifies the generated code statically before it merges. Multi-agent review cross-checks one model’s output with a different model and a different prompt, so you don’t depend on a single model’s stability (Sonnet 87%, Haiku 73% on the Part 3 n=3 numbers). Next to it sits static analysis, plain rule-based pattern detection. Missing volatile, ISR blocking, error-path cleanup, the 49-day wrap: the anti-patterns grep can find.
What static analysis misses goes to a deterministic analyzer. Keep board, errata, contract, and requirement as git YAML facts, and give the same facts to both the LLM at generation time and the CI gate. Defects that static analysis misses for lack of board context get caught here: eMMC wear accumulation patterns, board-specific contract violations, board-fact checks like “this counter is 32-bit, so it wraps.” When this layer works, even a category like SDMA comes down to Tier 2.
The last gate in the Review layer is human expert review. With the automated stages in place ahead of it, expert time goes to system-level calls: architectural fit, novel defects, domain trade-offs.
The Test layer verifies behavior by running it. With TDD the test cases come before the code, which narrows the space an LLM would fill with hallucination. On-board test is HIL with one real board wired into CI, and it catches what QEMU can’t see: DMA cache coherency, interrupt latency, clock jitter.
Tool choices at each stage, how deep to go in each layer, and how information flows between layers are for the next part.
Limits
3-Tier Trust is a classification and nothing more. Classification by itself catches no defects. It has to run together with the verification gates. Board Profile, datasheet RAG, a HIL farm: this context engineering work is the infrastructure those gates stand on.
EmbedEval is about 81% Zephyr. On ESP-IDF or STM32 HAL the Tier boundaries may shift. In the end you have to measure the numbers for your own platform.
The last limit weighs the most. For a team without experts, Tier classification is no answer. With nobody to write Tier 3 from scratch, you cannot even tell whether the LLM output is any good. In that team, adopting an LLM grows risk more than it grows productivity. Part 1’s dead eMMC is the proof. The Tier classification did not prevent the accident; it only put a label on it afterwards.
Wrapping Up
Part 1 was the case: LLM-written code can kill an eMMC. Part 2 was the taxonomy: that failure resolves into six patterns, clustered in Tier 3. Part 3 measured in numbers where the boundary sits. Part 4 carries that measurement into the big picture of operating rules, as an overview of the verification gate pipeline. Taking the gates apart one at a time, by tooling, configuration, and trade-off, I leave to the next part.
The SDMA fix on i.MX8M is one line of dma_alloc_coherent(). But the judgment to write that line came from the reference manual and the oscilloscope. It did not come from tokens. Looking back, the whole series comes down to that one sentence.
Comments
Loading comments...