← Essays /Post · 22 of 28 · Methods

AI Finally Needs Hardware Engineers

In 2026 inference overtook training. The edge is where it grows fastest, and the work left there, I think, is the work AI cannot do.

·5 min read · · · #edge-ai#inference#on-device
AI Finally Needs Hardware Engineers
On this page

In 2023, training took two-thirds of all AI compute. By 2026, inference had taken that place. In exactly three years the ratio turned over. Most of that inference still runs in the cloud. But the part growing fastest is the edge, and the edge is where embedded engineers work.

The reason, I think, is simple. Inference runs where the data is. On the device, at the edge, on the hardware we build.

What Converged

Three things happened at once. Any one of them alone would probably have fallen short.

Cost collapsed. When GPT-4 shipped in March 2023, inference ran $30-60 per million tokens. Today the same quality costs under $1. That is a 50x drop in three years. On-device, the gap grows wider still. A Jetson AGX runs a single inference at roughly $0.004, while a cloud API call runs $0.05-0.50. Production deployments in manufacturing and healthcare keep reporting 10-30x savings after moving on-device. When inference gets cheaper, deployments grow. More deployments mean more edge devices, and more edge devices mean more need for the engineers who keep them running.

Small models got good. Gemma 3, at 270 million parameters, runs on a Raspberry Pi 5. It is not fast. It is fast enough, though, for work where latency is measured in seconds, like batch entity extraction or classification. The breakthrough did not come from compression. It came from how the models were trained: below 1 billion parameters, distillation from large teacher models and high-quality synthetic data turned out to matter more than raw parameter count. Models that fit our hardware are no longer toy demos.

NPU silicon became standard. Right now I’m comparing the STM32N6 and NXP MCX N94 for a sensor gateway project. Both carry NPUs. Choosing the NPU variant adds less than a dollar to the BOM. A year ago that chip wasn’t on Mouser. Two years ago, “an NPU on an MCU” was something you saw in conference demos. AI acceleration has come down from differentiator to checkbox, the same path WiFi took on microcontrollers a decade ago.

What the Numbers Hide

Every vendor puts TOPS up front. I read it as the new MHz marketing. The real bottleneck on an edge device is memory bandwidth. Mobile sits at 60-85 GB/s (LPDDR5x); datacenter GPUs sit at 3,000-8,000 GB/s (HBM3/HBM3e). That 50-100x gap is what sets LLM decode speed at the edge. Bandwidth decides it more than raw compute does. This is also why 4-bit quantization moves edge throughput so much. Looked at closely, it is more a bandwidth problem than a size problem.

The tooling is not mature yet either. I lost a full day converting a keyword-spotting model from ONNX to LiteRT for an STM32N6 target. The model used depthwise separable convolutions, the standard architecture for efficient audio models. The STM32N6 NPU backend didn’t support the depthwise conv operator. In the end I rewrote the architecture around grouped standard convolutions. The model ran. The day was gone.

That was one project. Still, any embedded engineer who has put a model on an NPU probably has a story like it. The silicon works, and the framework has holes. It is not so different from the ESP8266 era. The hardware shipped, and the software is still catching up. There is one difference. WiFi settled on a single stack, while edge AI has dozens of frameworks competing and no winner yet. With things this split, this phase could last longer than WiFi’s did.

The honest architecture of most production systems shows this. 78% of production AI deployments run a hybrid edge/cloud setup. Pure on-device is closer to the exception, showing up only in offline, air-gapped (I have built one such pipeline), or hard-latency cases. Hybrid does not make the edge side any simpler, though. You decide what runs locally and what goes to the cloud, tune the on-device portion for latency and power, and handle the fallback when the link drops. That is systems engineering.

My Read on This

AI can generate code, propose architectures, and tune hyperparameters. It will get better at all of it. But inference at the edge does not end as a pure software problem. It sits on the boundary between the digital and the physical, and on that boundary, I think, AI stops being useful and starts needing us.

Sometimes NPU inference comes back at 200ms instead of 20ms. Then someone has to sit with the oscilloscope and trace the bottleneck to memory bandwidth on a specific bus, not to compute. When the product has to ship on a coin cell, someone has to choose what to give up between model accuracy and the power budget. Someone has to measure, verify, and own the result. In embedded, “plausible” and “correct” are not the same thing, and a wrong register address fails silently.

The more code AI produces, the more the person who can verify it on real hardware is worth. The more tradeoffs AI proposes, the more the person who knows which tradeoff fits a specific product, market, and power ceiling is worth. There are the constraints we always grumbled about: memory limits, thermal budgets, certification requirements, BOM pressure. Looking back, those were the moat. Not every product needs edge inference. Plenty will keep calling cloud APIs, and that’s fine. But for products that have to run offline, or live under a power budget, or hit hard latency targets, there is no shortcut around the hardware.

This is what I do now. I default to NPU-equipped SoCs (the BOM difference is noise), but I judge them by operator coverage rather than TOPS. I set aside time in the project schedule for adapting the model architecture. The distance between “this NPU supports INT8” and “this NPU supports my model’s operators” is usually a week of rework, and no datasheet tells you that up front. And I think the engineers who map this gap between silicon and software early will build the edge AI products that lead. It is the same pattern as the engineers who tamed the early ESP8266 WiFi stack and then built the IoT products that came after.

Inference won. What is left is the work AI cannot do: making things run correctly on constrained hardware, under real conditions, where failure has real consequences. This window will not stay open forever. Automated instrumentation and AI-driven testing will slowly take over parts of this space. The advantage goes to the engineers who move now.

ecro
Written by ecro

Created an LLM benchmark for firmware. EmbedEval →

Building a project-shaped agent harness for Claude Code, Cursor, and Codex. harness-maker →

Building a serial and SSH terminal where coding agents work on the board beside you. NeuroTerm →

Comments

Loading comments...