Workspace proxy (47083) hangs while the control API stays up; webviews and workspace verbs all fail, no way to restart just the proxy

Overview Files Skills Discover Packages Releases Dependencies Issues20 Breadcrumbs Pull Requests Activity

Open bug report

John Lauer · 2d ago

What happens

On AdomLapper (Windows, WSL2 container), Hydrogen's workspace proxy on 127.0.0.1:47083 stopped answering, while the control API on 47084 still responds in about 2 ms.

curl -m 8 http://127.0.0.1:47083/containers/current times out (connect deadline elapsed). GET /health on 47084 returns 200.
GET /diagnostics/all on 47084 fails three checks: adom-cli-api ("Carbon API unreachable: GET http://127.0.0.1:47083/health"), sse and webview (both "Cannot reach ... 127.0.0.1:47083/api/workspaces/editor/hdlocal/project/current").
~/.adom/hydrogen-ports.json matches the live ports (control 47084, proxy 47083, cdp 47085), so this is not the stale-ports case.
Everything that goes through the proxy is down: adom-cli hydrogen workspace tabs, opening or refreshing a webview tab, and loading any <port>.localhost:47083 app. An app's show verb (adom-aiflow-robotarm) failed the same way.
The control API's unknown-route listing only shows about 39 container/setup/claude endpoints, with nothing to restart or repair the proxy, so the only fix available to an AI is a full Hydrogen relaunch, which interrupts the person's work.

Expected

The supervisor should notice the dead proxy (the diagnostics already do) and restart just the proxy, or expose POST /proxy/restart on the control API.

Seen 2026-10-20 around 19:40 CT.

1 Reply

John Lauer · 1d ago

Root cause found and fixed on main; it ships in 1.0.661 and stays open until the promotion gate passes on that build.

What happened. The PC ran out of a Windows kernel resource: an Armoury Crate service (asus_framework.exe) had leaked about 35,000 handles, the nonpaged pool was at 1.6 GB, and Windows answered ERROR_NO_SYSTEM_RESOURCES (1450). tokio's I/O driver panics on any poll error except Interrupted, so a worker thread died at 19:16 and took the proxy's accept loop with it. The control API happened to survive on another worker. Nothing supervised the proxy, so :47083 refused connections until a manual restart. POST /app/restart then answered ok but never restarted, because it scheduled the restart on an async timer that never fired.

Fixes (commits ea0dfd0c, 757a07c9, d579fbd5):

Root: Hydrogen now builds with a vendored tokio 1.53.1 (src-tauri/vendor/tokio, see ADOM-PATCH.md). Its I/O driver backs off 50 ms and retries on transient resource errors (1450 and relatives, WSAENOBUFS, OutOfMemory) instead of panicking. gate_tokio_io_driver_patched fails to compile if the patch is ever dropped.
Proxy self-heal: an OS-thread watchdog probes the proxy port every 15 s and rebinds it after two misses. If three rebinds fail, it restarts Hydrogen. Accept errors no longer end the listener.
POST /proxy/restart (what this issue asked for): it stops the listener, and the watchdog rebinds it within about 35 s.
Control API watchdog: if :47084 stops answering for about 45 s, Hydrogen restarts itself. Watchdog restarts are limited to one per 10 minutes, so a port held by another program can't cause a loop.
/app/restart and /app/relaunch-frontend now schedule the restart on an OS thread, so a sick runtime can't swallow it.
Starvation warning: every 5 minutes Hydrogen samples the nonpaged pool and the biggest handle holder. At 1 GB or more it shows a toast naming the leaking program ("Armoury Crate (asus_framework.exe) is holding 34,973 handles..."), so the user isn't left thinking Hydrogen is broken.

New gate checks: proxy-self-heal, tokio-resource-exhaustion, control-api-watchdog, starvation-warning. The gate's unit run now includes hydrogen-proxy too; before this it never ran those tests.

AdomLapper was restarted by hand at 21:43 CT and the proxy is serving again.