Slow bridge /health is reported as a failed spawn, then locked in backoff with no clear cause

by adom

Adom Bridge unleashes your AI onto your full PC: full power, full safety. The AI breaks out of the container it runs in and onto your real machine, managing and running your entire laptop as you: launch apps, move files, drive any window, control KiCad, Fusion 360 or your real signed-in browser. Works with any AI, cloud or local (Claude Desktop, Claude Code, Codex), no Hydrogen required. Bridge and its bridges are an operating system for AI, with managed Node and Python runtimes, and a human-onl

Overview Files Skills Discover Packages Releases Dependencies Issues20 Breadcrumbs Pull Requests1 Activity

Closed bug report

John Lauer · 2mo ago ·closed by John Lauer

ab 2.0.84 · adom/core 4.13.14 · rustdesk bridge 0.3.2 (at time of failure) · Windows 11 Home build 26200 · Python 3.14.0

Summary

A bridge whose health endpoint is merely SLOW is treated as a failed spawn, and after enough retries the bridge is locked into spawn-backoff with no way to run any of its verbs. The surfaced error names the symptom ("did not respond") but nothing tells the caller that a health-check TIMEOUT was the cause, so it reads like a crashed or broken bridge.

This cost a long debugging session on a real user-facing outage (RustDesk remote access was unusable). The bridge process was healthy the whole time; it just answered /health too slowly.

What happened

rustdesk_connect and every other rustdesk_* verb failed with:

Failed to spawn bridge 'rustdesk': bridge 'rustdesk' is backing off after 16 failed spawn attempt(s): bridge 'rustdesk' did not respond on http://127.0.0.1:557...

The bridge's own log showed ab hanging up mid-response:

File "...\bridges-cache\rustdesk\server.py", line 368, in do_GET
    self._send(200, status_payload())
File "...\bridges-cache\rustdesk\server.py", line 361, in _send
    self.wfile.write(body)
File "C:\Python314\Lib\socketserver.py", line 845, in write
    self._sock.sendall(b)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

Root cause (in the bridge, now fixed in rustdesk-bridge 0.3.4): its status_payload() shelled out to tasklist /V on every /health poll. Measured on this machine:

tasklist_V_ms=2957
tasklist_noV_ms=3556

~3s per health call, which exceeded ab's spawn health-check timeout. ab aborted the connection, counted the spawn as failed, retried, and after 16-17 attempts entered backoff.

Proof it was only latency, never a broken bridge: run by hand, the exact same server.py starts and serves fine ->

rustdesk bridge v0.3.2 listening on 127.0.0.1:59999 (idle-disconnect 3600s)

After caching the session list on a background thread (rustdesk-bridge 0.3.4), the same endpoint answers in 22-74ms and ab spawns it first try.

What I expected

That a slow-but-alive bridge is distinguishable from a dead one, and that the error tells me which one I have.

Suggestions (any one of these would have saved the session)

Name the timeout in the error. "did not respond" should say "did not respond within Ns (health-check timeout)" and report the measured latency. Right now the caller cannot tell "timed out" from "crashed" from "wrong port".
Log slow health responses as a warning before failing. If /health answers but takes > ~1s, emit a warning naming the bridge and the latency. That single line would have pointed straight at the cause.
Do not count a TIMEOUT toward permanent backoff the same way a crash does, or make backoff clearable without restarting ab. Today the only recovery I found was restarting Adom Bridge entirely; bridge_kill returned error and refresh_bridges / rustdesk_readiness just timed out, because everything routes through the wedged spawn path.
Surface the health-check budget in the SDK docs so bridge authors know the ceiling they must stay under. The bridge SDK says the health endpoint should be cheap, but not what "cheap" means numerically.

Note

The bridge-side bug is already fixed and published (adom/rustdesk-bridge 0.3.4). This issue is only about ab's diagnosability and backoff recovery, which would help any bridge author who trips the same wire.

1 Reply

John Lauer · 2mo ago

Fixed in 2.0.84, and you found a real logic bug, not just a diagnosability gap. Thank you for the measurements - tasklist_V_ms=2957 is what made this a ten-minute diagnosis instead of a hunt.

The cause. ab's health probe timeout was 2 s. Your /health took ~3 s, so every probe timed out. And a timeout was classified Unreachable - the identical value ab uses for connection-refused. So a live bridge and a dead one were the same thing to ab: the spawn was scored a failure, retried, and after 16 attempts locked into backoff while your process sat there healthy. That is why the error said "did not respond" - ab genuinely could not tell.

What changed:

HealthProbe::TimedOut(ms) is its own arm, carrying the measured wait. A timeout is no longer collapsed into "unreachable".
The budget is now 6 s (was 2 s). It is a ceiling, not a floor - a dead port refuses instantly, so a genuinely absent bridge still fails fast.
A spawn whose probes timed out now SUCCEEDS, with a loud warning. The socket accepting the connection proves the bridge is up; scoring that a failed spawn is what took you down. Your bridge would have stayed up on 0.3.2.
The "dead" message now says it is not a timeout: "the port never accepted a connection, so nothing is listening. This is NOT a slow-health timeout; ab reports those separately and treats them as up."
A health answer past 1 s logs a one-time warning naming the latency (once per bridge - the loop runs every 15 s and would otherwise bury the log). That is your ask 2.
The numbers are published at status.bridgeHealth: probeTimeoutSecs, slowWarnMs, pollIntervalSecs, spawnGraceSecs. That is your ask 4 - they existed only as Rust constants, which is why you could not find them. A contract nobody can read is not a contract.

On your ask 3 (backoff recovery): partly addressed - a timeout no longer counts toward backoff, which removes the way you hit it. A general "clear backoff without restarting ab" lever is still missing and I have not built it; if you hit backoff another way, say so and I will.

Your 0.3.4 background-thread fix is still the right call - 22-74 ms is well inside the budget, and staying there means ab never has to guess about you.

John Lauer closed this · 2mo ago