01The situation
The devices sit behind NAT in places I can't reach directly. The only way in is through WireGuard gateways, and an update meant an SSH session per device, done by hand, one at a time.
I built a tool that updates a whole batch in one run, shows progress live, recovers from failures on its own and leaves a record of everything it did.
02What it had to do
- No direct access to devices: every connection goes through a gateway.
- Safe to interrupt: a stopped run must never leave a device worse off.
- Every action recorded, per device, so you can answer "what happened to device X?".
- Secrets never on the download server and never in a process argument.
03Architecture
Operator workstation
fleet-update CLI
Plans the batch, runs up to 30 devices at once and shows each one's stage live.
Results picker
After a run: logs, restart, roll back or retry, per device.
Audit log
Append-only JSON lines with session and device IDs, beside full per-device logs.
Downward link: SSH over WireGuard, one multiplexed session per gateway
Access
WireGuard gateways
The only way to reach a device. Checked in order: recent handshake, SSH port, ping; the next gateway takes over if one can't reach it.
One SSH connection per gateway
Multiplexed, so a hardware key prompts once per gateway, not once per device.
Downward link: SSH through the gateway to each device, up to 30 in parallel
Device
Updater
Download, verify, back up, install atomically, then health-check; roll back on failure.
Health watchdog
Paused during the install and restored by an exit trap on every path, Ctrl-C included.
App containers
Matched by label after start; each must be running, not just created.
Upward link: The device pulls the release over HTTPS and checks every file against the manifest
Release server
Versioned artifacts
Each release with a SHA-256 manifest the updater checks against.
No secrets
Credentials are passed at run time and never stored here.
Every step writes an audit event: session, device, action and result.
A device is only ever on the old version or the new one: installs are atomic and a write-once backup is kept for rollback.
Stopping a run is safe: devices mid-install finish that step, the rest are left untouched.
04How it works
Find a route
For each device, pick a gateway and check the device is online: a recent WireGuard handshake first, then an SSH port probe, then ping. If that gateway can't reach it, fall through to the next.
Update in parallel
Up to 30 devices at once, over one shared SSH connection per gateway, so a hardware key prompts once per gateway instead of once per device. A live dashboard shows every device's stage.
Verify before installing
On the device, the updater checks each downloaded file against a published SHA-256, keeps a write-once backup of what it replaces, installs atomically, then checks services and containers came back.
Keep the watchdog alive
The updater pauses the device's health watchdog while it works. An exit trap restores it on any exit, including a failure or an operator pressing Ctrl-C.
Act on the results
After a batch, a picker lists every device touched with its result. From there: view logs, restart the stack, roll back to the backup, or retry just the failures.
Record everything
Each action goes into an append-only JSON-lines audit log with session and device IDs, next to full per-device logs.
05Run it yourself
A simulated run of the tool on a made-up fleet: 24 devices, three gateways with one down, six at a time. Nothing here touches a real host; the names, versions and timings are invented.
fleet-update · demo fleet · 24 devices · 6 at a time · 2.3.1 → 2.4.0
14:02:00- gw-a up
- gw-b up
- gw-c down
0 updated24 queued
Run it, then try Stop halfway: devices mid-install finish their atomic step, the rest are left untouched, and every device ends on a known version.
device log
Nothing has run yet.
audit.jsonl0 events
Append-only. One line per action.
06What I found along the way
Healthy-looking devices with corrupt files
The device filesystem held writes in memory for minutes before committing them. A power cut soon after an update could leave service files corrupted while the device kept running fine from memory. Flushing data to disk before each atomic rename closed the gap.
"PASS" with a dead stack
Starting containers reports success once they're created, even if they crash a second later. I added a post-update check that matches containers by label and confirms each one is actually running.
Aborts left devices unwatched
A failed update stopped the watchdog and never restarted it, and nothing noticed because the watchdog is what would have noticed. The exit trap now restores it on every path, and Ctrl-C becomes a normal exit so the trap runs.
The bottleneck wasn't where it looked
Timing every run showed container image pulls dominated each update, and the registry, not the tool, was the limit, so adding concurrency only made it worse. I recommended a CDN or pull-through cache.
07What changed
- A fleet update is now a single command with a live view, instead of a manual SSH session to every device.
- Failures land in a re-runnable list, and any device can be rolled back from the same menu.
- Four defects that made devices look healthy when they weren't were found and fixed in the updater.
08Takeaways
- Verify on hardware, not by reasoning: kill a test update mid-pull and watch what really comes back.
- Measure before tuning. The fix for slow updates was at the origin, not in the scheduler.