Embedded devices need updates - bug fixes, security patches, new features. But updating firmware is risky. A failed update can brick a device, and bricked devices in the field mean truck rolls, recalls, or angry customers. Reliable update strategies ensure devices remain functional even when things go wrong.
What Can Go Wrong
- Power failure: Device loses power mid-update. Partially written firmware won't boot.
- Corrupt download: Network errors corrupt the firmware image.
- Bad firmware: The new firmware has a bug that prevents boot.
- Incompatible update: New firmware doesn't work with device hardware revision.
- Security compromise: Attacker installs malicious firmware.
The cardinal rule: A device must always be able to recover to a working state. If an update fails, the device should boot the previous firmware, not become a brick.
A/B Partition Strategy
The most reliable approach uses two firmware partitions. The device boots from partition A while updates are written to partition B. After a successful update, the device switches to partition B. If the new firmware fails, it can fall back to partition A.
How It Works
- Device running firmware from partition A
- New firmware downloaded and written to partition B
- Bootloader flags partition B as "pending verification"
- Device reboots into partition B
- If boot successful, partition B marked as "active"
- If boot fails, watchdog triggers reboot into partition A
Benefits
- Power-safe: Power loss during download doesn't affect running system.
- Automatic rollback: Failed updates revert without user intervention.
- Fast rollback: No re-download needed - old firmware still present.
- No downtime during download: Device keeps running while update writes.
Trade-offs
- Requires 2x storage for firmware images
- More complex bootloader logic
- Data migration between firmware versions needs handling
Recovery Partition Strategy
An alternative for storage-constrained devices. A small recovery partition contains minimal firmware that can download and flash the main firmware.
How It Works
- Device runs main firmware
- Update downloaded to staging area
- Device reboots into recovery partition
- Recovery flashes update to main partition
- Device reboots into updated main firmware
- If main fails, recovery can re-download and try again
Recovery Requirements
- Recovery partition must never be updated
- Recovery needs network capability to download firmware
- Recovery should be minimal - less code means fewer bugs
Secure Updates
Firmware updates are an attack vector. Without security, attackers can install malicious firmware. Essential security measures:
Code Signing
All firmware images are cryptographically signed. The bootloader verifies signatures before accepting updates. Only firmware signed with your private key will be installed.
Signing Best Practices
- Keep signing keys in HSM (Hardware Security Module)
- Use separate keys for development and production
- Implement key rotation capability
- Sign entire firmware image, not just parts
Secure Boot Chain
Each stage of boot verifies the next. ROM bootloader verifies first-stage bootloader, which verifies second-stage, which verifies firmware. If any stage fails verification, boot stops.
Encrypted Updates
Encrypt firmware images to prevent reverse engineering and protect intellectual property. Each device decrypts using its unique key.
Transport Security
Downloads should use TLS. Certificate pinning prevents man-in-the-middle attacks. Verify server identity before downloading.
Rollout Strategies
Staged Rollout
Don't push updates to all devices at once. Roll out to a small percentage first, monitor for problems, then gradually expand. If issues appear, stop the rollout before affecting the entire fleet.
- 5% of devices for initial testing
- 25% after 24 hours if no issues
- 100% after another 24-48 hours
Canary Updates
Target specific devices for early updates - internal test devices, cooperative customers, or devices with telemetry enabled. Learn from canaries before broad deployment.
Device Groups
Group devices by hardware revision, configuration, or customer. Update groups independently, applying updates only to compatible devices.
Update Monitoring
Know what's happening during updates. Track:
- Update status: Pending, downloading, installing, verified, failed
- Version distribution: What percentage on each firmware version
- Failure rates: Download failures, verification failures, boot failures
- Rollback events: How many devices fell back to previous firmware
High failure rates indicate problems with the update or with specific device populations. Investigate before continuing rollout.
Summary
Reliable firmware updates require multiple layers: resilient update mechanisms (A/B partitioning), security (code signing, secure boot), and operational controls (staged rollout, monitoring). The goal is zero bricked devices - every device should recover from any failure.
Design update capability from the start. Retrofitting reliable updates into a device designed without them is difficult. The extra storage and bootloader complexity is worth it when you have thousands of devices in the field.