Closed general

Relay `serve` has no supervision in containers → recurring 500 reconnect loops; add a keepalive convention/skill

Drew Owens · 18d ago ·closed by John Lauer

Problem (hit live, 2026-07-23)

A container that runs adom-desktop serve to stay connected into AD has no supervision for the relay process. When serve dies (e.g. the Claude Code session that launched it is torn down), the container's /proxy/8765/ endpoint has no upstream and returns HTTP 500 on every WebSocket upgrade. Every connected desktop AD then falls into an endless Connection error: HTTP 500 → Reconnecting in 30s loop.

Observed on the AdomPNP dev container: the relay had died overnight; the Essemtec machine's AD (essemtec-PC4203) and a laptop both 500-looped for hours until the relay was manually restarted. A sibling container (gallia) with a live relay was unaffected, which is the tell that this is per-container relay death, not a gateway problem.

Why it recurs

Service containers (factory-backups, factoryos) self-heal because they ship a watchdog cron. But a default dev container has neither crontab nor a usable systemd (systemctl is-system-running → offline, PID 1 is sh), so there is no native supervisor. An agent that runs adom-desktop serve to bridge the container into AD has no obvious, portable way to keep it alive, and typically doesn't realize it needs to.

Ask

  1. Ship supervision in AD itself — e.g. adom-desktop serve --supervise (or a serve that self-restarts / writes a small keepalive), so a single documented command survives the launching session. This is the robust fix and removes the footgun entirely.
  2. Failing that, a skill/guidance convention: when an agent runs adom-desktop serve in a container, it should also install a keepalive that works even without cron/systemd (a setsid-detached supervisor loop reparented to init is the portable fallback). Surfacing this in the adom-desktop / adom-desktop-discovery skill ("if you connect a container into AD, keep the relay alive like this") would make agents do it by default.

Workaround in place

Detached supervisor loop (~/adom-relay-keepalive.sh, setsid, 60s check on 127.0.0.1:8766/health, relaunch if down). Proven: killing the relay revives it in ~10s. Caveat: without cron there is no @reboot, so it does not survive a full container restart — which is exactly why native supervision (option 1) is preferable.

2 Replies

John Lauer · 18d ago

Shipped option 1 in v1.9.168adom-desktop serve --supervise. Thank you for the diagnosis; the root cause (session-child relay + no cron/systemd to self-heal) and the setsid pointer were exactly right, and it's what the fix is built on.

What it does

serve --supervise re-launches a watchdog detached into its own session via setsid, reparented to init — so the launching shell/Claude-Code session going away no longer takes the relay down. Verified in the process table: the watchdog comes up with PPID=1 and its own SID. That's the core fix; the rest is the safety net.

The watchdog then keeps the relay alive, but deliberately not as a blind restart loop:

  • Exponential backoff with a cap — 2s → 4s → … → 5 min, and a clean run (relay stayed up ≥60s) resets it. So if the relay is genuinely broken it's never bludgeoned-restarted; the retries just space out.
  • It reports every restart — a restart means something is wrong, so once the relay is healthy again it fires an AD toast to the connected desktops with the reason pulled from ~/.adom/relay.log. So the owner hears about a crash-loop instead of it silently thrashing. --owner <clientName> targets one desktop; default is all connected.
  • serve --stop stops the watchdog + relay cleanly (clears the supervisor pidfile first so it can't restart in the race).

Plain serve now also prints a loud hint on launch that it will die with the session and points at --supervise — the self-aware "you're about to footgun yourself" nudge. And the skills (main adom-desktop + adom-desktop-discovery) now recommend serve --supervise in containers and explain why serve & is the trap, so agents do it by default.

On your reboot caveat — you're right and I didn't try to paper over it: --supervise covers session teardown (the failure you actually hit), but a full container reboot still needs a boot hook (cron @reboot / the container's init), which a session-launched command can't provide. The launch output says so explicitly. Your ~/adom-relay-keepalive.sh workaround can retire for the session-teardown case; keep a boot hook if you want reboot-survival too.

To get it: adom-wiki pkg update (or reinstall adom/adom-desktop), then adom-desktop --version ≥ 1.9.168. In a container: adom-desktop serve --supervise (and serve --stop to stop it).

John Lauer · 18d ago

Follow-up — an important gap in what I said above (John caught it): 1.9.168's --supervise fixes the session-teardown death, but it did NOT handle the pkg update interaction, and one part was a regression. Both are fixed now in 1.9.169.

What was wrong:

  • A pkg update replaces the binary via unlink+recreate. So the running relay kept executing the OLD (now-deleted) inode — adom-desktop --version reported the new version while the relay ran old code (silent divergence).
  • Worse, the watchdog respawned the relay via current_exe(), which after an update points at the deleted inode — so a restart after an update hit ENOENT and the watchdog could never bring the relay back (proven live). That's worse than no watchdog.

Fixed in 1.9.169:

  • The watchdog resolves the binary via PATH (the ~/.local/bin/adom-desktop symlink that tracks the current install) on every spawn, with a current_exe() fallback that strips a (deleted) suffix — so a restart never bricks and naturally picks up the new binary.
  • It also detects a newer on-disk version and gracefully restarts the relay into it, toasting "auto-updated X -> Y". So the relay stays alive AND current across updates.

Verified on Linux: after a binary swap + relay kill, respawn in ~3s (was a permanent brick); and a version bump on disk triggers a clean auto-restart. pkg update to 1.9.169.

Log in to reply.