From TinyML to Tiny Language Models: the State of Edge AI in 2026

There is a small pile of circuit boards on my desk, and between them they tell you most of what has happened in applied edge machine learning over the last few years. One of them costs about €35 and recognises spoken keywords using a microphone and a neural network, with no internet connection anywhere in my home. Another is a camera the size of a matchbox that runs a modern object detector at thirty frames a second while drawing about a quarter of a watt. A third is a Raspberry Pi with a €70 accelerator attached, and it will happily run a compact language model, or detect people in a video stream at speeds that required a server-class GPU not long ago. The most expensive item in the pile is a victim of the recent datacentre demand pricing for DDR RAM, but the price is still tolerable.

This blog article is a short survey of how that became possible and where it stands in mid-2026: what "edge AI" actually spans, which models are state of the art on which class of device, what the neural accelerators inside these boards really do, and how federated learning lets fleets of devices improve without ever pooling their data. This is a reading article rather than a building one – there is not a line of code in it. The building happens across the rest of this blog and in future articles. The final section maps those hands-on articles onto the landscape described here. If this piece does its job, you will finish it wanting to build something, knowing roughly which device to reach for.

Why intelligence is leaving the cloud

For a decade, the story of machine learning was a story about scale: bigger models, bigger datasets, bigger datacentres. That story is still true at the frontier. But a second story has been running quietly alongside it, and for anyone who works with sensors, robots, wearables or buildings, it is arguably the more consequential one – the models are becoming small and efficient enough to leave the datacentre entirely.

The motivation is easiest to see by counting what a cloud round-trip actually costs. Suppose a sensor captures an image, ships it to a cloud API, and receives a label back. That loop pays five separate embedded-world "taxes":

• Latency: Tens to hundreds of milliseconds each way, with the variance mattering more than the mean. A fall detector, an industrial robot guardrail or a self-navigating drone cannot wait out a slow round-trip, and some control loops simply cannot close through a WAN at all.
• Privacy: The raw data leaves the building. For audio and video from homes, offices, hospitals and classrooms, that is a serious commitment, and under legal requirements such as the GDPR and the EU AI Act it is increasingly a legal question rather than a design preference question. Data that is never transmitted is data that cannot be intercepted or eves dropped in transit.
• Bandwidth and cost: Video is data heavy. A single camera streaming continuously can consume more upstream bandwidth than the rest of a household combined, and cloud inference is metered; per-call pricing that looks negligible in a demo could be incredibly expensive at one call per second per device for years.
• Resilience: A device that needs the cloud to think stops thinking when the network fails, which is precisely when monitoring systems tend to matter most. Self-navigating car pedestrian detection systems cannot be allowed to fail if the network fails.
• Energy: Moving data costs energy. For small payloads and small models, the radio can spend more energy transmitting a measurement than the processor would spend analysing it locally, a trade that battery-powered design (as in my sleepy Thread sensor) revolves around.

Edge inference answers all five at once: run the model where the data is born, transmit only conclusions. An occupancy sensor that sends "room occupied, confidence 0.94" instead of a video stream has transformed its privacy posture, its bandwidth bill and its failure modes in one move.

It is worth being clear about what edge inference does not answer. No microcontroller will host a frontier model, and tasks that need broad world knowledge, long reasoning chains or up-to-date information still belong upstream. The pattern the industry has largely settled on for 2026 is hybrid: local models handle the routine, latency-sensitive, privacy-sensitive work, and the hard or rare cases are escalated to larger models in the cloud (a division of labour that industry analysts now present as the default architecture rather than an exotic choice; see the Edge AI and Vision Alliance's review of on-device LLMs for a practitioner's account). The interesting engineering question is no longer whether to put intelligence at the edge, but how much fits at each price and power point. Answering that requires a short history and then a guide.

How we got here: a short history of small machine learning

The DSP era (before roughly 2017). Signal processing on microcontrollers is old; "machine learning" on them mostly meant hand-tuned filters, thresholds and the occasional decision tree. Neural networks were things that lived on GPUs.

TinyML (roughly 2017 to 2022). Three developments converged. Google's team, needing "OK Google" detection to run continuously on phone hardware, showed that useful keyword-spotting networks could fit in tens of kilobytes; Pete Warden's "Why the future of machine learning is tiny" became the movement's manifesto, and the TinyML book its textbook. TensorFlow Lite for Microcontrollers (now part of Google's LiteRT family) provided a runtime small enough for devices with kilobytes, not gigabytes, of memory. And research such as MIT's MCUNet showed that co-designing the network architecture with the memory constraints of a microcontroller could put genuine ImageNet-class vision, at reduced resolution and accuracy, on a Cortex-M. Platforms such as Edge Impulse then packaged the whole pipeline (data collection, training, quantisation, deployment) into something a student could complete in an afternoon. The characteristic TinyML workload was perception: is a keyword present, which gesture was that, is there a person in this frame.

The NPU (roughly 2022 to 2025). TinyML ran its networks on the CPU, helped by vector instructions and libraries such as CMSIS-NN. The next step was architectural: dedicated neural processing units (NPUs) appeared not just in phones but inside microcontrollers. Arm's Ethos-U55 microNPU, which you will meet on my desk shortly, was designed to sit beside a Cortex-M and multiply its inference throughput by an order of magnitude or more within a milliwatt-class budget. STMicroelectronics built its own Neural-ART accelerator into the STM32N6, the first STM32 with an NPU. Discrete accelerator modules, most visibly Hailo's M.2 cards for single-board computers, did the same for the Linux tier. Neural acceleration stopped being exotic silicon and became a standard feature.

The small language model era (2024 onwards). The surprise of the last two years is that the compression techniques sharpened on perception models turned out to work on generative ones. A research and product race documented in the survey literature produced small language models (SLMs) in the hundreds-of-millions to few-billions of parameters range that are genuinely useful: Microsoft's Phi series demonstrated that data quality could substitute for sheer size, Google's Gemma line brought per-layer embedding tricks that shrink the memory footprint of its edge-focused variants, and Alibaba's Qwen small series ships sizes from 0.8B parameters explicitly aimed at IoT-class hardware. Quantised to four bits, a capable ~4B-parameter model now occupies close to a gigabyte and runs on a phone, a Raspberry Pi 5, or an accelerator module drawing a few watts. Language, summarisation, structured extraction and simple reasoning joined perception on the list of things a device can do by itself.

The result of these three waves stacking is the ladder we can now climb.

A guide to the edge: the compute ladder

"Edge" is used so loosely that it can mean anything from a smartwatch to a rack in a factory. It clarifies enormously if you arrange the space by power budget, because power predicts almost everything else: cost, memory, and what class of model fits. Here is the ladder as it stands in 2026, with the rungs illustrated by the boards used throughout this blog (prices approximate):

Three observations about the ladder are worth making explicit, because they shape every project decision:

• Each rung is roughly an order of magnitude in power and compute, and the model families sort themselves accordingly. Knowing your power budget usually tells you your model class before any benchmarking happens.
• The bottom rungs are not obsolete. It is tempting to read the ladder as a progress narrative in which everything migrates upward. The opposite is true: a plain Cortex-M4F drawing microwatts asleep is the only rung that runs for a year on a coin cell, and "is anyone in this room" often needs a 20 KB network, not a 20 TOPS accelerator. State of the art at the bottom rung means state of the art efficiency, and it is advancing as quickly as the top.
• Sensors themselves are climbing the ladder. The RD-03D radar module featured in my FMCW radar article runs its own tracking algorithm on the module and emits target coordinates, not raw chirp data. Smart sensors that pre-digest reality are edge AI too, and pairing them with a modest MCU often beats pointing a big model at raw data.

What actually runs there: state-of-the-art models in 2026

Perception: mature, cheap, everywhere

The perception workloads that defined TinyML are now thoroughly commoditised. Keyword spotting in a few tens of kilobytes; gesture and activity recognition from IMU streams; visual person detection in a couple of hundred kilobytes; acoustic and vibration anomaly detection for machinery. On the vision side, compact members of the YOLO family remain the workhorses for detection on accelerated hardware (YOLOv8n is the model I benchmark across five platforms in this blog's capstone), while architectures such as MobileNetV4 represent the state of the art in efficient classification backbones, designed explicitly to run well across CPUs, DSPs and NPUs alike. For speech-to-text, distilled variants of OpenAI's Whisper bring usable transcription to SBC-class devices (a feature that is welcome to Home Assistant enthusiasts).

If there is a headline at this tier, it is that transformer architectures have arrived at the microcontroller. Attention-based models were long considered datacentre only; steady work on operator support in runtimes and on NPU instruction sets (Arm's Ethos-U85, below, supports transformer operators natively) means compact vision transformers and attention-augmented detectors are now realistic on milliwatt-class silicon. The practical effect is not that everyone runs ViTs on MCUs; it is that the architectural gap between "what research publishes" and "what an MCU can execute" has narrowed dramatically.

Language and multimodality: the SLM shift

The most visible change since this blog's earlier series is that language models became a realistic edge workload. As of mid-2026 the names to know are:

• Gemma (Google): the edge-focused variants (the current generation's "E"-designated models, around 2B and 4B effective parameters) use per-layer embeddings to cut runtime memory well below what the parameter count suggests, and are the reference targets for Google's LiteRT-LM on-device inference stack.
• Phi (Microsoft): the Phi-4-mini class (~3.8B parameters) continues the series' signature trait of outsized reasoning benchmark scores for its size, making it a favourite where the edge task involves structured thinking rather than open-ended chat.
• Qwen small series (Alibaba): a spread of sizes from 0.8B to 9B parameters released in early 2026, notable for explicitly targeting deployment tiers from IoT devices up to local servers with one model family.

The enabling arithmetic is quantisation. Weights trained in 16-bit floating point are compressed to 4-bit integers using techniques (AWQ and GPTQ are the common ones) that choose which weights can least afford the precision loss; a ~4B model shrinks from ~8 GB to roughly 2 GB while retaining the large majority of its benchmark performance. That is the difference between "needs a GPU" and "fits beside the browser tabs on a Raspberry Pi 5". Runtimes such as llama.cpp made this tier accessible to hobbyists years before the vendor stacks arrived, and remain the bridge for running open SLMs on CPUs.

On the ladder, language capability sorts like this in 2026: at the plain-MCU rung, none (and be suspicious of anyone claiming otherwise); at the SBC-CPU rung, ~1B models generate at usable-but-slow rates; with a generative-capable accelerator (Hailo-10H class, with its own on-module DRAM) or a phone NPU, 2-4B models run at conversational speed within single-digit watts. The sensible edge applications are correspondingly modest and correspondingly valuable: summarising a day of sensor events in a sentence, turning natural-language commands into device actions, extracting structure from messy local text, all without a byte leaving the premises.

The compression toolbox

Whatever the model class, the same four techniques carry it down the ladder, and they are worth knowing by name because every deployment tool you meet is some packaging of them:

• Quantisation: represent weights (and often activations) in fewer bits. INT8 is the common language of edge accelerators; INT4 is now routine for language model weights. This is performed post-training or, for the last few points of accuracy, during training (quantisation-aware training).
• Pruning: remove weights or whole channels that contribute little, trading a small accuracy loss for size and speed.
• Knowledge distillation: train a small "student" model to imitate a large "teacher", capturing more capability per parameter than training small from scratch. Most named SLMs lean on this.
• Architecture search and co-design: design the network for the target hardware's memory and operator constraints from the start, the MCUNet insight that now pervades the field.

The order is not critical: at the edge, the model is a workhorse component to be engineered against a budget, not an artefact to be admired.

Hardware Acceleration Demystified

It is easy to treat an NPU as a magic sticker on the datasheet. The underlying story is more instructive, and it explains both why the accelerators work and why their headline numbers mislead.

A neural network's arithmetic is overwhelmingly multiply-accumulate (MAC) operations arranged in regular patterns. A general-purpose CPU spends most of its silicon on being ready for any code: caches, branch predictors, out-of-order machinery. An NPU inverts that approach: it has a large array of MAC units and just enough control to push tensors through them. Regularity is key, and it is also the first caveat: an NPU accelerates the operators it implements, and a model containing anything else falls back to the CPU, which is why "does my model's operator set map onto this accelerator" is the first real question of any edge deployment.

The second design force is the memory wall. Performing a MAC costs almost nothing; fetching its operands from off-chip memory costs orders of magnitude more energy. Every serious edge accelerator is therefore, at heart, a scheme for moving data less: local SRAM tiles beside the MAC arrays, weight compression, and compilers that tile and reorder the computation to maximise reuse. When an accelerator vendor quotes TOPS-per-watt figures that seem implausible next to a GPU, data locality is where the difference lives.

Three concrete implementations, all present on my desk:

• Arm Ethos-U55 (in the OpenMV AE3, paired with Cortex-M55 cores): a microNPU, configurable from 32 to 256 MAC units, that sits beside a microcontroller core and executes quantised INT8 networks compiled ahead of time by the Vela compiler. Its successor, the Ethos-U85, scales to 2048 MACs (up to 4 TOPS at 1 GHz), adds native support for transformer operators, and signals where the microcontroller tier goes next.
• ST Neural-ART (in the OpenMV N6's STM32N6): ST's in-house NPU, roughly 300 MAC units at 1 GHz for about 600 GOPS at around 3 TOPS/W, fed by ST's own compiler toolchain. Its significance is less the number than the precedent: a mainstream MCU vendor deciding NPUs belong in the standard portfolio.
• Hailo-8L (on the Pi 5 AI Kit): a discrete dataflow accelerator, 13 TOPS INT8, that decomposes a network into a pipeline laid out spatially across its compute fabric so that activations stream between layers without round-tripping to memory. The newer Hailo-10H adds on-module LPDDR4 (the missing part for models whose weights dwarf on-chip memory), which is what makes 2-4B-parameter generative models feasible at the SBC tier within a few watts.

And a note of caution the industry has earned: TOPS is a capacity, not a speed. The figure states how many operations the MAC arrays could theoretically complete, at a chosen precision (a 40 TOPS INT4 figure is not comparable to a 13 TOPS INT8 one), saying nothing about whether your model's operators, shapes and memory traffic can keep the arrays fed. Utilisation on real networks varies enormously, which is why independent benchmark suites such as MLPerf Tiny exist, and why this blog's capstone article runs the same YOLOv8n network on five platforms and reports measured frames per second and energy per inference rather than datasheet arithmetic. Treat TOPS as a hint about which rung of the ladder a part belongs to, and measure everything else.

Federated learning: training without collecting

Everything so far concerns inference: models trained elsewhere, executed locally. But a fleet of edge devices is also, collectively, the owner of a dataset that no one else has, and often one that nobody is permitted to centralise: audio inside homes, health signals from wearables, telemetry from machines belonging to competing customers. Federated learning (FL) is the family of techniques for training on that data without gathering it.

The canonical algorithm, federated averaging (FedAvg, from the 2017 paper that founded the field), is relatively simple to state. A coordinating server holds the current global model. Each round, it sends that model to a sample of devices; each device trains for a few steps on its own local data; each sends back only the resulting weight update, never the data; the server averages the updates into a new global model and repeats. Google's early production deployment, improving keyboard predictions in Gboard, remains the clearest existence proof: millions of phones collectively improving a model from data (everything people type) that could never ethically be uploaded.

The elegant sketch hides real difficulties:

• Statistical heterogeneity. Each device sees a biased slice of reality (your keyboard is not my keyboard; a radar in a gym does not see what a radar in a library sees), and naive averaging of models trained on wildly different distributions can converge slowly or badly. A large literature of FedAvg refinements exists to manage this.
• System heterogeneity. Devices differ in compute, memory, connectivity and availability; stragglers and dropouts are the norm. Practical FL systems sample participants and tolerate partial rounds.
• Communication cost. Weight updates for even small deep models are megabytes; for battery- and bandwidth-constrained devices, update compression and infrequent participation are essential, and for microcontroller-class devices this cost is one of the two blocking constraints.
• Privacy is improved, not solved. Weight updates can leak information about training data, so serious deployments layer on secure aggregation (cryptographic protocols, such as Bonawitz et al.'s, that let the server learn only the sum of updates, never any individual one) and differential privacy (calibrated noise that bounds what any single user's data can reveal). These are standard equipment in production FL, not academic dressing.

On tooling, the ecosystem has consolidated pleasantly. Flower (which began at Oxford; the framework paper is a readable introduction) has become the de facto open framework: it is agnostic about the underlying ML library, runs on anything from a workstation to a fleet of Raspberry Pis, and its simulation mode lets you develop and debug a federation on one machine before touching hardware, a workflow philosophically identical to this blog's habit of building simulators next to firmware.

Where does FL sit on our compute ladder? Frankly: training, even fine-tuning, is far more demanding than inference, so in 2026 cross-device FL is practical from roughly the Raspberry Pi rung upward, and thoroughly proven at phone class. On microcontrollers it remains a research frontier (the reviews of FL on edge sensing devices survey the attempts): tiny models can be fine-tuned on-device, but full FL participation strains flash write endurance, RAM and radio budgets simultaneously. The pragmatic near-term architecture for an MCU fleet is a layered one, and it is worth noticing that this blog has already built its skeleton: sleepy sensors report compact readings over Thread to a hub (as in the CoAP fleet hub), and it is the hub tier, Pi-class and mains-powered, that participates in federation on the fleet's behalf. Each building's hub learns from its own sensors; buildings share model updates, not sensor data.

Two adjacent ideas complete the picture. On-device personalisation skips the federation and simply fine-tunes a small model locally (increasingly via parameter-efficient methods such as LoRA, where only small adapter matrices are trained), so your wake-word detector adapts to your voice without any of it leaving the device; combining local personalisation with federated global improvement is the pattern behind modern keyboards and assistants. Split learning divides a model between device and server so that neither raw data nor the full model crosses the link; it trades privacy properties for a tighter coupling to connectivity, and remains more niche.

The AIoT pattern: what all this makes possible

"AIoT" (artificial intelligence of things) is a marketing word, but it names a real architectural pattern, and it is the pattern this article has been assembling piece by piece:

1. Sense with rich or pre-digesting sensors (a camera, an IMU, a radar that tracks targets on-module).
2. Infer locally, converting raw signals into small, meaningful, privacy-preserving facts at the point of capture.
3. Connect frugally, moving those facts (not the signals) over low-power networking to wherever decisions are made.
4. Learn collectively, improving the fleet's models from the fleet's experience without centralising the fleet's data.

What is new in 2026 is not any single stage; it is that every stage is now within reach of the hardware in that pile on my desk, and of a university lab budget. Some concrete shapes the pattern takes:

• Occupancy and presence, without cameras. An mmWave radar plus a milliwatt-class inference model gives you room-level presence, count and even fall detection with nothing that resembles an image anywhere in the system, which changes the conversation entirely for homes, hospitals and hotel rooms. (The radar half of this is exactly the RD-03D work elsewhere on this blog; the Thread occupancy node in progress joins the pieces.)
• Sensor fusion that cancels each modality's lies. Cameras are fooled by posters (remember the infamous Mark Rober Tesla Looney Tunes Test!); radars are fooled by fans. A fusion state machine that requires agreement to enter "occupied" and tolerates disagreement to remain there (the design of my radar-camera fusion article) is edge AI at the systems level, where the intelligence lies in how cheap inferences are combined.
• Predictive maintenance. An accelerometer and a small autoencoder on a motor housing, flagging anomalous vibration spectra locally for pennies of hardware, is quietly one of the highest-value TinyML deployments in industry.
• Fleets that get smarter together. The endgame that federated learning unlocks: a hundred buildings' occupancy hubs, each locally adapted, collectively training better models with no central pool of anyone's movements. This is the layer this blog has not built yet, and the one the survey above suggests is now feasible with commodity parts and open frameworks.

The honest hard parts

A survey that only lists capabilities is more an advertisement. Five difficulties bite in 2026, and knowing them in advance is most of the defence:

• Toolchain fragmentation. Every NPU family has its own compiler and quantisation pipeline (Vela for Ethos-U, ST's tools for Neural-ART, Hailo's dataflow compiler, and so on), each with its own operator coverage. The model is portable; the deployment is not. Budget real time for the compile-quantise-validate loop on each target, and prefer models already proven on your silicon.
• The model lifecycle. Models are firmware now: they ship with bugs, they drift as the world changes, and they need secure updates for years. Everything this blog covers about OTA updates with MCUboot applies, with the added twist that a model update can silently change behaviour in ways a normal code diff never shows. Version your models, log their confidence distributions, and keep a rollback path.
• Evaluation and drift. The benchmark accuracy that chose your model was measured on someone else's data. The gap between that number and behaviour on your deployment's data, lighting, accents and machines is where edge projects quietly fail; local evaluation sets, however small, beat leaderboard faith.
• Security of the model itself. A model in device flash can be extracted (protect it as you would any IP), and models can be manipulated through their inputs by adversarial examples. Neither risk is a reason for paralysis; both are reasons the sensor-fusion habit of never trusting a single inference has security value as well as reliability value.
• Energy accounting. Datasheet TOPS-per-watt and real system power differ by everything else on the board: sensor supplies, memory, radios, idle floors. The only defensible numbers come from measuring whole-system energy per useful inference, which is why a power analyser features so prominently in this blog's benchmarking plans.

A curriculum, not a conclusion

The purpose of this article was to draw the map; the rest of this blog is the territory, walked with hardware you can buy for the price of a good meal out. If the survey above has a single takeaway, it is that every layer of the AIoT pattern is now accessible to an individual with modest kit, and here is how the articles on this site line up against it:

• The plain-MCU rung and its craft: the Zephyr and Embassy Rust foundations, and TinyML proper with keyword spotting and IMU gesture recognition on the Nano 33 BLE Sense (in progress).
• The microNPU rung: vision on the OpenMV N6's Neural-ART and low-power vision on the AE3's Ethos-U55s.
• The accelerated-SBC rung: the Pi 5 vision pipeline measured honestly, CPU against Hailo-8L.
• The receipts: the capstone, one network (YOLOv8n) measured five ways for frames, watts and euros.
• The connective tissue: Thread sensors, CoAP fleets, the radar work, and the occupancy node that fuses them.
• The learning layer: federated learning across the fleet's hubs is the natural next series once the sensing and inference layers are validated; consider this article its statement of intent.

Start anywhere on the ladder; the rungs connect.

Further reading

• A Survey of Small Language Models (Nguyen et al.): the academic map of the SLM space.
• On-Device LLMs in 2026 (Edge AI and Vision Alliance): a practitioner's view of the generative edge.
• MCUNet: Tiny Deep Learning on IoT Devices (Lin et al.): the paper that made microcontroller vision credible.
• Communication-Efficient Learning of Deep Networks from Decentralized Data (McMahan et al.): the FedAvg paper; still the right first read on federated learning.
• Flower: A Friendly Federated Learning Framework (Beutel et al.), and the Flower documentation.
• Federated Learning on Edge Sensing Devices: A Review: where FL meets the constrained tiers of the ladder.
• Arm Ethos-U85 announcement and ST's Neural-ART introduction: the microNPU tier in the vendors' own words.
• MLPerf Tiny: independent benchmarking for the smallest rung.
• TinyML (Warden and Situnayake): dated in its tooling, timeless in its constraints-first mindset.