AI Finally Needs Hardware Engineers

In 2023, training ate two-thirds of all AI compute. By 2026, inference had taken that spot. The ratio flipped in exactly three years. Most of that inference still runs in the cloud. But the fastest-growing slice is the edge, and the edge is where embedded engineers work.

The reason is simple. Inference, by definition, runs where the data is. On the device. At the edge. On the hardware we build.

What Converged

Three things happened at once. Any one of them alone would have fallen short.

Cost collapsed. When GPT-4 shipped in March 2023, inference ran $30-60 per million tokens. Today the same quality costs under $1. A 50x drop in three years. On-device the gap widens further. A Jetson AGX runs a single inference at roughly $0.004, while a cloud API call runs $0.05-0.50. Production deployments in manufacturing and healthcare keep reporting 10-30x savings after moving on-device. Cheaper inference means more deployments. More deployments means more edge devices, and more edge devices means more demand for the engineers who keep them running.

Small models got good. Gemma 3, at 270 million parameters, runs on a Raspberry Pi 5. Not fast. But fast enough for work measured in seconds, like batch entity extraction or classification. The breakthrough had nothing to do with compression. It came down to training methodology: distillation from large teacher models and high-quality synthetic data turned out to matter more than raw parameter count below 1 billion. Models that fit our hardware stopped being toy demos.

NPU silicon became standard. Right now I’m comparing the STM32N6 and NXP MCX N94 for a sensor gateway project. Both carry NPUs. The NPU variant adds less than a dollar to the BOM. A year ago that chip wasn’t on Mouser. Two years ago, “an NPU on an MCU” was a conference demo. AI acceleration slid from differentiator to checkbox, the exact path WiFi took on microcontrollers a decade ago.

What the Numbers Hide

Every vendor leads with TOPS. It’s the new MHz marketing. The real bottleneck on an edge device is memory bandwidth. Mobile sits at 60-85 GB/s (LPDDR5x); datacenter GPUs sit at 3,000-8,000 GB/s (HBM3/HBM3e). That 50-100x gap is what governs LLM decode speed at the edge. Bandwidth decides it, not raw compute. This is also why 4-bit quantization hits edge throughput so hard. At bottom it’s a bandwidth problem more than a size problem.

And the tooling is still green. I lost a full day converting a keyword-spotting model from ONNX to LiteRT for an STM32N6 target. The model used depthwise separable convolutions, the standard architecture for efficient audio models. The STM32N6 NPU backend didn’t support the depthwise conv operator. So I rewrote the architecture around grouped standard convolutions instead. The model ran. The day was gone.

That was one project. But any embedded engineer who has put a model on an NPU has a version of this story. The silicon works; the framework has holes. It’s the ESP8266 era all over again. The hardware shipped, the software is still catching up. The one difference: WiFi converged on a single stack. Edge AI has dozens of frameworks fighting it out with no winner yet. That fragmentation could stretch this phase longer than WiFi’s.

The honest architecture of most production systems reflects exactly this. 78% of production AI deployments run a hybrid edge/cloud setup. Pure on-device stays the exception, showing up only in offline, air-gapped (I have built one such pipeline), or hard-latency cases. Hybrid doesn’t make the edge side any simpler, though. Deciding what runs locally and what goes to the cloud, tuning the on-device portion for latency and power, handling the fallback when the link drops. That’s systems engineering.

My Read on This

AI can generate code, propose architectures, tune hyperparameters. It will get better at all of it. But inference at the edge doesn’t stay a pure software problem. It sits on the boundary between the digital and the physical, and on that boundary AI stops being useful and starts needing us.

When NPU inference comes back at 200ms instead of 20ms, someone has to sit with the oscilloscope and trace the bottleneck to memory bandwidth on a specific bus rather than to compute. When the product has to ship on a coin cell, someone has to choose what gives between model accuracy and the power budget. Someone has to measure, verify, and own the result. In embedded, “plausible” and “correct” aren’t the same thing, and a wrong register address fails silently.

The more code AI produces, the more valuable the person who can verify it on real hardware becomes. The more tradeoffs AI proposes, the more valuable the person who knows which tradeoff fits a specific product, market, and power ceiling becomes. The constraints we always grumbled about. Memory limits, thermal budgets, certification requirements, BOM pressure. Those turned out to be the moat. Not every product needs edge inference. Plenty will keep calling cloud APIs, and that’s fine. But for the products that have to run offline, or live under a power budget, or hit hard latency targets, there’s no shortcut around the hardware.

Here’s what I do now. I default to NPU-equipped SoCs (the BOM difference is noise), but I evaluate them by operator coverage rather than TOPS. I bake model-architecture adaptation time into the project schedule, because the gap between “this NPU supports INT8” and “this NPU supports my model’s operators” is usually a week of rework, and no datasheet warns you up front. And I’m betting the engineers who map this silicon-to-software gap early will build the edge AI products that dominate. Same pattern as the engineers who tamed the early ESP8266 WiFi stack and then built the IoT products that came after.

Inference won. What remains is the work AI cannot do: making it run correctly on constrained hardware, under real conditions, where failure carries real consequences. This window won’t stay open forever. Automated instrumentation and AI-driven testing will chip away at this space. The advantage goes to the engineers who move now.