app

Adom Hydrogen

Public Made by Adom

by adom

The whole Adom experience as one signed native app: your editor, your AI agent, and the full electronics design toolchain, running locally on your own machine.

Overview Files Skills Packages Releases Dependencies Issues11 Breadcrumbs Pull Requests1 Activity
Open general

Workspace keeps dropping ('CONTAINER STOPPED' / 'Cannot reconnect, reload window'): false-negative health probe forces code-server restarts; code-server is actually up (HTTP 200, NRestarts=0)

Drew Owens · 7d ago

Type: Bug report

Symptom: The workspace keeps dropping. HD shows "CONTAINER STOPPED" and pops "Cannot reconnect. Please reload the window." The user reloaded 5 times in about 30 minutes (2026-08-03, HD session started 09:26:51).

Diagnosis: the disconnects are self-inflicted by a false-negative health probe. code-server and the distro are fine.

What I verified in the container while HD was calling it dead:

  • code-server answers instantly from inside the distro: curl localhost:7380/healthz returns HTTP 200 in 3 ms; the extension on :8821 returns 200.
  • systemd reports code-server active (running) with NRestarts=0. It never crashed on its own. Its only restarts are the ones HD forced (last forced start 10:04:18).
  • WSL networkingMode=mirrored is enabled (HD logs it at startup), so host to distro loopback is available.

What HD's health monitor did instead, today, in this one session:

  • Declared the workspace unhealthy 136 times, every one reading code-server not responding (http=Some(false) ...). The http=Some(false) means HD's own host-side HTTP probe to code-server failed, while code-server was answering 200 from inside.
  • wsl_alive flips between true and false across consecutive probes seconds apart, so HD's WSL liveness check (host wsl.exe) is also flapping.
  • Forced 7 bounded code-server restarts (rate-limited to 1 per 5 min). Each forced restart plus each unhealthy verdict is what tears the iframe's connection and raises "Cannot reconnect, reload the window." The remediation is more disruptive than the transient it thinks it sees.

The tell that this is HD's prober, not the services: at the exact same timestamps, HD's AD health probe is also failing and respawning Adom Desktop every ~10 seconds:

10:04:17 [wsl] health monitor: restarting code-server (bounded)
10:04:20 [ad-supervisor] AD health probe failed - attempting respawn -> pid 92112
10:04:22 [wsl] Workspace unhealthy — code-server not responding (http=Some(false))
10:04:30 [ad-supervisor] AD health probe failed - attempting respawn -> pid 47176

Two independent targets (code-server inside the distro, AD on the Windows host) failing the same probe in lockstep is a prober fault, not two services dying together. The host was not starved either: the status bar read CPU 1 percent, RAM 6 percent. Corroboration: my adom-desktop calls to the host started returning empty mid-investigation, consistent with AD being respawned (breakaway-detached) out from under the relay every 10 seconds.

Likely root cause (hypothesis): HD's health-probe path is unreliable. The wsl.exe-based liveness check flaps (it is known to wedge, serialize behind a global lock, and throw E_UNEXPECTED), and the host to distro HTTP probe intermittently returns false under whatever is also driving the concurrent AD-probe failures. This looks related to the AD respawn churn and the ~20 Hz relay reconnect a parallel investigation is tracking: HD respawning AD in a loop would produce exactly that relay churn and log storm.

Asks:

  1. Stop acting on single false negatives. Require several consecutive failed probes with a sane timeout, and confirm from inside the distro (a real request to localhost:7380) before declaring the workspace down. code-server answering 200 in 3 ms must not read as "not responding."
  2. Make the remediation less destructive than the fault. A forced code-server restart and a "reload the window" prompt drop the user's session. Do not trigger them on a flapping probe; that is the actual cause of the 5 reloads.
  3. Fix the shared prober. code-server and AD probes failing together points at HD's probe mechanism or the host to distro channel, not at two services. Treat this and the AD respawn or relay churn as one root cause.
  4. Log probe results with the real error (timeout vs refused vs wsl error) and rate-limit the repeats. 136 identical unhealthy lines in one session is noise that hides the signal.

Env: HD build 458c322 era, code-server 1.124.2, distro Adom-Workspace, WSL mirrored networking on. Full HD log saved.

2 Replies

John Lauer · 7d ago

Excellent report, and the lockstep observation (code-server probe + AD probe failing together) was the key. Status: hardening is code-complete on main (commit 32b75245) and ships in the next build; keeping this OPEN until it is validated on the affected machine.

What landed, mapped to your asks:

  1. No more single-probe verdicts: workspace_stats now requires THREE consecutive failed rounds. While debouncing it reports "unknown" so the UI keeps its previous state (no more CONTAINER STOPPED flash on a transient refuse).
  2. Confirm from inside before remediating: at three strikes, if the host loopback refused, HD runs the health check INSIDE the distro. If code-server answers healthy there, the forced restart is SUPPRESSED (it would not fix a host-side path fault and it is what tears the session) and the log names the real suspect: "host loopback REFUSES :7380 but code-server answers healthy INSIDE the distro - remediation suppressed, host port-forward flap suspected".
  3. Shared-prober fault treated as one thing: the AD supervisor got the same 3-strike debounce; a single failed 5s probe no longer respawns AD. The ~10s respawn churn you correlated (and which broke your own adom-desktop calls mid-investigation) cannot recur from one flapping probe.
  4. Honest, bounded logging: unhealthy lines now carry the consecutive count and whether the inside-check vouched; the 30s rate limit stays.

Still open here: the underlying host-loopback flap itself (WHY 127.0.0.1:7380 intermittently refuses while mirrored networking is up) is not yet root-caused; the hardening makes it non-destructive and clearly logged so the next occurrence produces a clean signature instead of 136 identical lines.

John Lauer · 6d ago

2026-08-04: major hardening shipped around the stop/start/wedge surface (dev builds 8f8b2f4..472e4759):

  • Stop Container now actually stops. USER_STOPPED latch owned by hd-control, enforced at BOTH exec choke points (hd-app run_wsl_timeout refuses -d execs while latched; hd-control wsl_exec_adom same). Three separate resurrection paths found and killed: the ~1/s stats probe, the reconnect-dialog watcher's health probe, and a post-restart async settings re-assert. Ralph-tested: distro stays Stopped across 110s after a stop that lands 50s after a restart, with refusal lines in the log.
  • WSL wedge self-heal ladder extended. HCS_E_CONNECTION_TIMEOUT now classified as a wedge; when wsl --shutdown + retry stays wedged, rung 2 kills stale wsl.exe CLIENT processes and retries (the manual recovery that worked live today).
  • Watcher log storm fixed (quiet:true on the internal 15-context walk): idle log rate 6-7 lines/min vs ~120+/min before.

Log in to reply.