Designing over-the-air updates that survive power loss, bad networks and bad releases

An over-the-air update system is not judged on the day it works. It is judged on the day the power goes out while a device is halfway through writing its kernel, the day a release ships with a bug that stops devices from phoning home, and the day a cellular network drops every download at 98 percent. If you design for those three days, the ordinary days take care of themselves.

This is the design we start from when a client asks for OTA, whether we end up building on Mender, RAUC, SWUpdate or something custom. It is split the way the responsibility is split: what the device must guarantee, what the server must guarantee, and how you test that both are true.

The device side: never be un-bootable

Two slots and a bootloader that can count

The core mechanism is dual-slot (A/B) storage for everything that has to be right for the device to boot: kernel, device tree, root filesystem, and usually the firmware for any co-processor. The running slot is never written. The update is written to the other slot, verified, and then the bootloader is told to try it once. The bootloader keeps a boot counter and a flag: if the new slot fails to boot and confirm itself within a bounded number of attempts, the bootloader switches back to the old slot on its own, with no help from software that may be broken.

With U-Boot this is the bootcount and upgrade_available mechanism, with the environment stored redundantly so that a power cut while saving it cannot corrupt both copies. With other bootloaders the names differ; the shape does not. The rule is that switching slots and rolling back must work when userspace is entirely broken, because a release that breaks userspace is exactly what you are protecting against.

Confirmation is a health check, not a boot

The new slot must confirm itself, and “the kernel started” is not confirmation. The update agent should mark the slot good only after the application-level checks that matter for your product have passed: the main service is running, the device can reach its cloud endpoint, the radio came up, storage mounted. Until then the boot counter keeps ticking. A release that boots to a login prompt but cannot talk to the cloud must roll back, or your fleet will go dark and stay dark.

Write safely, verify before you trust

• Stream to the slot. Do not download a full image to a temporary file and then copy it; most devices do not have the space, and it doubles the write wear. Stream, decompress and write in one pass.
• Verify the signature and hash before the slot is marked bootable. The signature check happens on the whole artifact; the hash of the written slot is checked after the write, because flash lies occasionally.
• Persist the update state machine. Downloading, written, verified, pending-reboot, confirming, committed: each transition is recorded in a small journal that survives power loss, so the agent knows where it was when it comes back.
• Make downloads resumable. HTTP range requests plus a stored offset. On cellular and satellite links a non-resumable 200 MB download is a download that never finishes.
• Ship deltas when bandwidth is the constraint. Block-based delta tools (casync, zchunk, librsync-style) cut typical updates by an order of magnitude; the cost is complexity in the build pipeline, not on the device.

Keep data out of the slots

Persistent data lives on its own partition, mounted by both slots. That means every release must be able to read the data format written by the previous one, and, because rollback exists, the previous release must tolerate data written by the new one. Schema migrations that only go forward are a rollback bug waiting to happen. Version the data format, migrate lazily, and test the downgrade path.

The server side: never brick the fleet at once

Signed metadata, not just signed images

Signing the image is necessary and not sufficient. An attacker who controls the update server, or a CDN, can still replay an old, vulnerable but validly signed image, or serve a mix of components that were never tested together. The answer, formalised in The Update Framework and its automotive descendant Uptane, is to sign the metadata: a manifest that lists exactly which artifacts, with which hashes and versions, constitute a release, with an expiry date and a monotonically increasing version number that devices refuse to go backwards on. Separate signing keys for the release manifest and for the artifacts limit what one compromised key can do. You do not need to adopt the whole framework to adopt the ideas.

Rollouts are a control loop

A release goes to a canary group first: a few percent of the fleet, chosen to cover hardware revisions and regions. The server watches the signals that matter, confirmation rate, rollback count, crash reports, heartbeat loss, and only widens the rollout when they hold. A halt criterion is defined before the rollout starts (for example, “more than two percent of updated devices roll back, or heartbeat loss exceeds baseline by one percent”) and the halt is automatic. Humans decide to resume; software decides to stop.

The device tells you what happened

Every device reports its current slot, current version, the last update attempt’s outcome and error, and the bootloader’s counter. Without that report the rollout control loop is blind. With it, “the update failed on 3 percent of Rev C boards with error verify-hash” is one query instead of a support escalation.

Audit everything

Who released what, to which group, when, and who halted it. Regulated industries require this; everyone else discovers they need it during the first incident review.

Build, buy or assemble

Three open-source projects already implement most of the device side above, and two of them the server side:

• Mender: A/B rootfs updates, delta updates, a device agent and a server (hosted or self-hosted) with rollout groups. The most complete end-to-end option; the server is where you pay, in licence or in operations.
• RAUC: a device-side update framework with a clean bundle format, slot model and bootloader integration for U-Boot, GRUB, Barebox and EFI. Pairs with Eclipse hawkBit for the server side.
• SWUpdate: device-side, very flexible handler model (raw images, files, scripts, container images), also commonly paired with hawkBit.

Custom makes sense when the constraints are unusual: a co-processor with its own update protocol, a bandwidth budget measured in kilobytes, a transport that is not HTTP, or a certification regime that requires you to own every line. Even then, copy the state machine and the slot model from the projects above rather than inventing them.

Testing the bad days on purpose

• Power-cut rig. A relay or USB power switch under test control cuts power at random points during download, write, verification and confirmation, thousands of times, and the device must come back on a bootable slot every time. This finds journal bugs, environment-save bugs and flash-driver bugs that nothing else finds.
• Network chaos. Drop the connection at random offsets, inject latency and loss, return HTTP errors mid-stream. Downloads must resume, not restart.
• Upgrade matrix. Every version you have ever shipped, updated to the candidate, then rolled back, then updated again. Data-format compatibility bugs live here.
• Bad-release drill. Ship a release that deliberately fails its health check to a canary group in staging and confirm that the rollout halts and every device rolls back without human intervention.