Adom Hydrogen

Follow-up: the wedged state was REAL this time, and that makes defect 2 worse — it is the clear that is wrong

Kept digging after filing, because the workspace never came back. Important correction to my own report: on this machine right now, WSL genuinely is wedged. Measured directly, outside Hydrogen:

wsl.exe --list --verbose            -> TIMED OUT at 15s
wsl.exe -d Adom-Workspace -- echo   -> TIMED OUT at 75s (elapsed 80.2s)

So the original report's framing ("WSL was never wedged, it was just a slow cosmetic probe") is only true of the earlier flap. The detector's declaration is now correct. What is still wrong — and is worse than I first described — is the clear.

Wsl/Service/CreateInstance/0x800705b4 (ERROR_TIMEOUT) is the concrete failure, from the log:

10:29:53.595 [wsl] systemd code-server setup failed (wsl ["-d", "Adom-Workspace", "-u", "root", "--", ...])
  exited with Some(-1): This operation returned because the timeout period expired.
  Error code: Wsl/Service/CreateInstance/0x800705b4), using held-child fallback

Why the flap is the real bug, not a cosmetic one

Hydrogen declares WSL_WEDGED, correctly, and says it is "suppressing non-essential wsl.exe calls". Then ~2s later some cheap call returns and it clears. Then 28s later it declares again. Over and over, ~90 times in this log.

The consequence is not just log noise: Hydrogen never actually stays in the degraded mode it just entered. Each clear re-enables the non-essential wsl.exe calls it had just decided to suppress, which it then fires into an already-jammed wslservice. The suppression that is supposed to let the service drain never gets to run for more than two seconds at a stretch.

So defect 2 should probably be re-prioritised: the declare threshold is defensible; the clear needs to be much stricter (require N consecutive successes, or a successful expensive call — not whichever cheap one happens to return), plus a minimum dwell time before clearing.

Recovery through ah's own API cannot get out of this state

Tried the documented recovery path end to end. None of it recovers, and two things actively mislead:

1. /container-restart latches user_stopped and leaves it latched when the start fails.

10:39:09.826 [control] Workspace restart requested via API, emitting restart event
10:39:09.850 [wsl] user stop requested, latching stopped state (suppresses stats/health probes so nothing auto-restarts the distro)
10:39:09.851 [cmd] $ wsl --terminate Adom-Workspace

A restart takes the user-stop latch. The terminate succeeds, the start then fails against the wedged service, and the workspace is left both stopped and latched — with /container-status reporting:

"narrative": "Workspace is stopped by the user. Start it from the Adom menu (or POST /container-start).",
"user_stopped": true

which is false — the user did not stop it, a restart did. Anyone (human or AI) triaging from that line concludes the user stopped their own workspace. Suggest a restart should either not take the user-stop latch at all, or clear it when the start half fails.

2. code-server started as HD-held child is logged while the distro is not running.

10:46:57.732 [wsl] code-server started as HD-held child on 0.0.0.0:7380

At that moment, and continuously after it, health reports:

"code_server_alive_in_distro": false,
"code_server_listening_in_distro": false,
"distro_running": false,
"status": "exited"

The held-child fallback reports success on spawn without confirming the distro came up or the listener bound. That success line is what sends you looking in the wrong place.

Sequence, for reproduction

1. Machine sleeps ~15 min; on wake [main-watchdog] MAIN THREAD BLOCKED for ~939s.
2. Concurrently, several VS Code WSL windows reconnect to an unrelated personal distro, saturating wslservice.
3. wslservice stops servicing CreateInstance within any timeout ah uses.
4. WSL_WEDGED flaps declare/clear every 30s indefinitely, never holding suppression.
5. /container-restart → latches user-stopped, terminates, fails to start.
6. /container-start ×2 → "start requested", ok:true, distro stays exited.
7. Only remedy left is wsl --shutdown (or restarting WslService), which also tears down the user's unrelated distros and any unsaved work in them.

Step 7 is the part worth designing for: the only exit from this state is a machine-wide WSL teardown that costs the user their other work. If ah can detect CreateInstance timeouts specifically, it could say so plainly and name that cost, rather than repeating a generic "a Windows restart clears this" paragraph every 30 seconds.

Ruled out while diagnosing

For anyone triaging a similar report — these were checked and are not involved:

• No golden-image migration. No --import / unregister / rootfs / bake activity anywhere in the log; Adom-Workspace is still registered: true, setup state complete, 21/21 open-welcome DONE.
• Not the 1.0.164 WSLg change. %USERPROFILE%\.wslconfig is untouched since Aug 11 and contains no guiApplications; GET /wsl/gui reports configured: null, enabled: true, so setup never wrote it (setup was already complete, so the step never re-ran on update).
• Not a stale app. 1.0.164 is current for the public tier.

Environment unchanged from the original report: ah 1.0.164 (c4613fbf-dirty) · ab 2.1.8 · Windows 11 10.0.26200.9168 · WSL 2.7.11.0.

Resolved on this machine by wsl --shutdown — and the recovery itself is evidence

Ran wsl --shutdown. Two things worth recording.

1. The shutdown itself took 70 seconds. That is a command that normally returns in a second or two, and it is the cleanest single measurement of how jammed wslservice was:

wsl.exe --shutdown    -> completed, no output, elapsed 70.1s

2. Everything recovered immediately afterwards, with no Hydrogen restart needed.

wsl --list --verbose  -> 0.7s   (was: TIMED OUT at 15s)

  NAME              STATE           VERSION
* Ubuntu-24.04      Running         2
  docker-desktop    Stopped         2
  Adom-Workspace    Running         2

Hydrogen picked the workspace back up on its own:

10:55:21 running=True cs_alive=True listening=True host_reachable=True EDITOR_WILL_LOAD=True
10:55:42 running=True cs_alive=True listening=True host_reachable=True EDITOR_WILL_LOAD=True
10:56:02 running=True cs_alive=True listening=True host_reachable=True EDITOR_WILL_LOAD=True
10:56:22 running=True cs_alive=True listening=True host_reachable=True EDITOR_WILL_LOAD=True
10:56:43 running=True cs_alive=True listening=True host_reachable=True EDITOR_WILL_LOAD=True
10:57:03 running=True cs_alive=True listening=True host_reachable=True EDITOR_WILL_LOAD=True

Stable across ~2 minutes. No reinstall, no re-setup, no distro re-import, no ah restart.

What this confirms about the diagnosis

Hydrogen's install, the Adom-Workspace distro, and its contents were never damaged. The entire outage was wslservice refusing CreateInstance, and every symptom above it — the flapping banner, the failed /container-start calls, the held-child fallback, the editor stuck on "Starting AI Environment" — was downstream of that one thing.

That is a good result for ah's recovery path: once WSL is healthy again, ah self-heals with no user action. The gap is entirely in the ~40 minutes before that, where ah had the information to say what was wrong and instead repeated a generic paragraph 90 times.

The concrete ask, now that the shape is confirmed

ah already distinguishes the failure in its own logs — it logged Wsl/Service/CreateInstance/0x800705b4 at 10:29:53. So it can tell "the management plane will not create instances" apart from "a probe was slow". Suggested behavior when CreateInstance times out repeatedly:

1. Latch the wedged state (per the hysteresis point above) instead of clearing it 2s later, so call-suppression actually holds and stops adding load.
2. Say the specific thing once, not the generic thing every 30s — something like: "WSL's management plane is not starting distros (CreateInstance timeout). wsl --shutdown fixes this. Note: that restarts all your WSL distros, so save work in other WSL windows first."
3. Name the cost. The remedy is machine-wide and takes down the user's unrelated distros. The current message says "A Windows restart (or restarting WslService) clears this" without mentioning that either option costs the user their other WSL sessions — and without mentioning wsl --shutdown, which is the cheapest of the three and the one that actually worked here.

Point 3 is the one I'd prioritise for users: the fix was 70 seconds away the whole time, and nothing in the UI pointed at it.

The other two defects from the follow-up stand and are unaffected by the recovery — the false user_stopped after a failed restart, and code-server started as HD-held child logging success while distro_running: false.

Partial fix shipped in ah 1.0.197 (insiders): the toast half is fixed, the flapping half is not.

What changed. The banner was sticky (duration 0) and its "toast already shown" latch was one-way, so once raised it stayed on screen telling the user to restart Windows even after WSL_WEDGED cleared, which it does by itself on the next successful wsl.exe call. It now retracts on the first healthy stats poll and re-arms for a genuinely new wedge, and the copy leads with "this usually clears itself within a couple of minutes, often after the laptop wakes", with restarting Windows as the fallback rather than the instruction.

What did not change, and is the substance of this issue: the declare/clear flapping itself. wsl_note_timeout declares on the third consecutive timeout and wsl_note_ok clears on the next success, with no hysteresis, so a wslservice that is merely slow (your own distro restarting, VS Code WSL windows reconnecting) still oscillates on a metronome and still writes the full forensic paragraph each time. Your 70-second wsl --shutdown is the right measurement of that: the plane was jammed, not broken, and a transient jam should not read as a fault.

Reasonable next step, not yet done: require the streak to hold for a duration rather than a count, log the declaration once per episode rather than per declaration, and treat a clear inside a short window as "still degraded" rather than recovery.

Triaged with John today: fix now, as one change with #76. The wedge flag will only trip when a real management call (a start or stop the user or the setup cascade asked for) hangs. The stats read stops using wsl.exe entirely (CPU and GPU come from Win32, RAM and the OOM counter come over HTTP from inside the VM), so it can never feed the detector, and the WSL service banner can never be painted by a slow but healthy VM again.

Shipped in Hydrogen 1.0.387, as one change with #76.

The flapping came from the cosmetic stats read: the poll spawned wsl.exe every ~10s to read /proc inside the distro, and when that read exceeded its 3s cap three times (easy on a busy or thrashing box) it tripped the WSL_WEDGED detector, which painted the "restart Windows" banner over a workspace that was answering fine over HTTP.

Now the meter reads CPU, RAM and the kernel OOM count from the adom-vscode extension over HTTP (GET /stats, 1.1.59). The stats read no longer touches wsl.exe at all, so it can never feed the wedge detector. WSL_WEDGED is now set only by a genuine start/stop management-plane hang, and cleared as before. Verified on arav-rog: the meter now reports stats_source "distro" over HTTP with real numbers and wsl_wedged false.