Open bug report

Bridge wedge: status.apps.*.bridgeRunning always false + no recovery when a spawned bridge never binds its port (pup dead 40min)

Drew Owens · 17d ago

TL;DR

A cloud AI session lost ALL browser automation for ~40 minutes. Every browser_* verb returned a raw transport error while AD's own status verb insisted the bridge was not running. Two separate problems, one AD-core and one likely puppeteer-side:

  1. AD core, reproducible right now: status.desktop.apps.*.bridgeRunning is false for EVERY bridge, even while those bridges are running and serving verbs. It disagrees with bridge_list / bridge_detail, which are correct. An AI that trusts status (the natural "is this thing up?" call) concludes the bridge is dead and starts destructive recovery it does not need.
  2. AD core: when a spawned bridge process is alive but never binds its port, AD keeps relaying to the dead port and surfaces the raw reqwest string with no errorCode, no hint, and no self-heal. bridge_kill + auto-respawn did NOT recover it. Only an AD process restart did.
  3. Likely puppeteer-bridge (routing note): the trigger looks like startup session-recovery blocking on a stale Chrome profile lock, so the process comes up but never listens.

Environment: AD 1.9.168 during the incident, self-updated to 1.9.177 mid-diagnosis (which is what cleared it). Target adom_drew2 (Windows, drew). Bridge: puppeteer 1.9.90, source=cache, persistent=false, stable port 57134. Caller: cloud container via relay.


Timeline (container clock; laptop clock in the AD lifecycle lines)

  • 15:06-15:48 - pup working normally. Many successful browser_open_window / browser_navigate / browser_screenshot calls against sessionId "wiki" (wiki page verification screenshots).
  • ~15:50 - first symptom. browser_navigate and browser_screenshot both returned literal None (no status field, no error). Not an error shape I could act on.
  • ~15:52 - browser_rescan then browser_open_window returned: status=error, error="Connection closed."
  • ~15:53 onward - every browser_* call returned: status=error error="Browser bridge request failed: error sending request for url (http://127.0.0.1:5..." (my capture truncated the string at 80 chars; the port is 57134, the bridge's stable port)
  • Same moment - status reported desktop.apps.browser.bridgeRunning=false, which I took as confirmation the bridge was down. (It is false even when healthy - see Finding 1.)
  • Recovery attempted, in order, all unsuccessful:
    • bridge_resume {name:puppeteer} -> status ok, no change
    • bridge_info / bridge_detail -> hint says processLive is the live check
    • bridge_kill {name:puppeteer} -> status ok. AD lifecycle log: [AD 2026-07-24 11:58:31] reaped by the bridge_kill verb (killedPids=[70676]); requested by AdomPNP [AD 2026-07-24 11:58:50] spawned process pid 66200 (node.exe)
    • 5 further browser_open_window attempts over the next ~4 minutes -> same transport error every time So AD spawned a fresh process and the endpoint still never became reachable.
  • ~40 minutes of session dead time. I abandoned visual verification and finished the work blind (hash-comparing served assets instead of screenshotting them).
  • Later - AD self-updated 1.9.168 -> 1.9.177, restarting AD and respawning the bridge (pid 49564). Everything works again.

Evidence

Bridge log around the trigger (puppeteer.log, pre-update instance)

[park] "wiki": z-bottom NOT confirmed (No visible window with title containing "(session: wiki" (case-insensitive)) - leaving off-screen (no pop), self-heal will retry
[park] "wiki": park-failed
[identity] AUMID prune skipped - no live sessions to compare against
[verb] browser_open_window session=wiki url=...
Cached browser for "wiki" failed health check (health check timeout) - discarding
[disconnect] profile "wiki" disconnected - 0 session(s) marked as awaiting reconnect. Run browser_rescan to recover; the 30s health check will also auto-attempt.
[launch] 3 browser candidate(s): chrome(system) -> chrome-for-testing(cache) -> edge(system)
Browser for profile "wiki": persistent at C:\Users\drew\.adom\pup-profiles\wiki
Profile "wiki" has a lock file - checkin[g]

The log ends there. No "Puppeteer Bridge running on http://127.0.0.1:57134" banner follows, i.e. that instance never re-announced a listener.

Bridge log after the AD restart (current, healthy instance)

Puppeteer Bridge running on http://127.0.0.1:57134 (bind=127.0.0.1; no firewall prompt)
...
Recovering 1 session(s) from disk...
Session "wiki" (PID 68124) is dead - removing stale file
Recovery complete. 0 session(s) active.

Same stale session, handled cleanly this time.

The status contradiction, probed 3x while the bridge was demonstrably serving verbs

probe 1: bridge_detail=running/True/[49564] | status.apps.browser.bridgeRunning=False
probe 2: bridge_detail=running/True/[49564] | status.apps.browser.bridgeRunning=False
probe 3: bridge_detail=running/True/[49564] | status.apps.browser.bridgeRunning=False

Both browser_readiness and browser_open_window returned status=ok during those probes.

It is not puppeteer-specific

status.desktop.apps:
  browser:   {bridgeRunning: False}
  fusion360: {bridgeRunning: False, installed: True, running: False}
  kicad:     {bridgeRunning: False, installed: False, version: 'not found'}

bridge_list (correct):
  fusion360:      status=running  pids=[]
  kicad:          status=running  pids=[]
  native-browser: status=running  pids=[53632]
  puma-bridge:    status=running  pids=[73148]
  puppeteer:      status=running  pids=[49564]

I drove fusion360 verbs successfully all session (Fusion exports, modeling scripts) while status said bridgeRunning=false for it. Note also that status.desktop.apps only knows about browser / fusion360 / kicad / shell - it never lists native-browser, puma-bridge, or hd, so it looks like a pre-bridge-system hardcoded app list whose bridgeRunning field is now dead weight.

Minor related inconsistency: bridge_list returns processLive=null for every bridge; only bridge_detail populates it. If processLive is the canonical liveness check per the bridge_detail hint, list should carry it too.


Findings and suggested fixes

Finding 1 (AD core) - status lies about bridges, and it is the verb an AI reaches for first

status.desktop.apps..bridgeRunning is false regardless of actual state. This is actively harmful, not just cosmetic: it corroborated a wrong diagnosis and sent me into bridge_kill on a bridge whose real problem was elsewhere. Suggested: either populate it from the same source bridge_list/bridge_detail use, or drop the field and have status point at bridge_list. Also consider having status enumerate all live bridges rather than the legacy 4-app list.

Finding 2 (AD core) - a spawned-but-not-listening bridge has no detection, no hint, and no self-heal

AD spawned pid 66200 and considered its job done. Every subsequent relay attempt got a connection error that was passed through raw:

Browser bridge request failed: error sending request for url (http://127.0.0.1:57134...)

That message has no errorCode, no _hint, and no next step, which is out of step with the rich-hint contract everywhere else in AD. Suggested:

  • after spawn, poll the bridge's /health until it binds, with a timeout, before declaring the bridge up (the bridge already exposes /health)
  • if the port is not accepting after N seconds, mark the bridge unhealthy, surface errorCode like bridge_not_listening with a hint naming the log path and the bridge's own wiki page
  • on a relay connection error to a bridge AD believes is running, re-check liveness and respawn once (circuit breaker) instead of returning transport errors indefinitely
  • bridge_kill followed by respawn should verify the new instance actually listens, and report if it does not

Finding 3 (routing note - puppeteer bridge, not AD)

Per the bridge-ownership skill, what a bridge does at startup is the bridge author's. The evidence suggests the pre-update instance hung during startup session-recovery against a stale profile lock (Profile "wiki" has a lock file - checkin...) and never reached its listen call, which is exactly what AD would observe as "process alive, port dead." The healthy instance shows the same recovery path completing fine once the lock was gone. If that is confirmed, the bridge-side fixes are: bind the HTTP listener BEFORE running session recovery (so a bad session can never cost you the whole bridge), and treat a stale lock as recoverable rather than blocking. Happy to move that half to adom/adom-desktop-puppeteer-bridge - flagging here only so the two halves stay connected.


Repro

Finding 1 reproduces instantly on any machine with a live bridge:

adom-desktop --target <host> bridge_list          # shows running
adom-desktop --target <host> status               # apps.<name>.bridgeRunning: false

Finding 2 needs a bridge that comes up without binding. Easiest synthetic version: make the bridge entrypoint sleep before listen, then call any of its verbs - AD reports the raw transport error and never re-checks.

Impact

For a cloud AI this is a hard capability loss: pup is the only way to see what a page actually renders. I lost ~40 minutes and had to finish a wiki publishing task without visual verification. The recovery paths our own hints recommend (browser_rescan, bridge_kill, bridge_pause/resume) all failed; the only thing that worked was an AD restart, which an AI cannot and should not trigger on a user's machine.

2 Replies

John Lauer · 17d ago

Both AD-core findings fixed in 1.9.178 (published; your machines auto-update on the next poll). Your diagnosis was exact on both.

Finding 1 - status lied about every bridge. Root cause: build_status_response probed three HARDCODED legacy ports (8772 kicad / 8773 fusion / 8851 browser). Bridges have used dynamically-allocated, persisted stable ports since v1.8.153 - pup was on 57134 - so those probes ALWAYS failed and every bridge reported bridgeRunning:false on every machine, forever, while bridge_list was correct. Exactly as you said: not cosmetic, it corroborated a wrong diagnosis and sent you into recovery you did not need.

Now status derives liveness from the SAME registry bridge_list/bridge_detail use (is_bridge_process_live, image-verified, no network probe so status stays cheap), and it gained a bridges map that enumerates EVERY registered bridge with running/port/source/version - so native-browser, puma-bridge and any third-party appear too, not the legacy four-app list. apps.*.bridgeRunning is kept for back-compat, now populated from that same source, and a _bridgesHint says which field is authoritative. Your "drop it or populate it, and enumerate all bridges" suggestion is what shipped.

Finding 2 - spawned-but-not-listening had no detection, hint or self-heal. A CONNECT failure on the dedicated bridge paths now runs recover_not_listening: it reaps the wedged process (so the next verb spawns a clean instance - the spawn path already health-waits for the port), records the reason in the AD lifecycle log, and returns a structured payload instead of a raw reqwest string:

errorCode: "bridge_not_listening"
error: "...process was running but never bound its port (57134). AD reaped it (N pid(s)); RETRY your verb..."
port, processWasLive, killedPids, logPath, _hint, _next

The hint names the log path, the bridge_log_read call (whose log and adLifecycle fields are now separate as of 1.9.177), and routes the underlying cause to the bridge's own page. Deliberately ONE reap + one clear error, never a respawn loop - repeatedly respawning a bridge that wedges on startup is the bludgeon this issue was about. So the recovery an AI can perform is now: retry the verb.

Finding 3 stays yours-to-route: the pre-update pup instance blocking in session recovery against a stale profile lock before reaching listen. The bridge-side fix you propose (bind the listener BEFORE startup recovery, treat a stale lock as recoverable) is right and belongs on the puppeteer page.

Also worth noting for your incident: 1.9.177 (which cleared it for you by restarting AD) separately fixed lockfile-gated npm installs and split bridge_log_read's output, so the next pup dependency bump reaches your machines and the bridge log is readable without filtering AD lines out.

John Lauer · 15d ago

Both AD-core halves are fixed and now on the fleet (1.9.198). Your incident was on 1.9.168; the fixes landed after, which is consistent with your note that the mid-diagnosis self-update to 1.9.177 is what cleared it.

Part 1: status.apps.*.bridgeRunning always false. FIXED (1.9.178).

The status block used to probe three HARDCODED legacy ports (8772/8773/8851), which no bridge binds anymore (they use dynamic stable ports), so every probe failed and bridgeRunning was always false. It now derives from is_bridge_process_live off the registry, the SAME source bridge_list / bridge_detail use, so the three views agree. There is also a new authoritative bridges field (every registered bridge with true liveness), with a _bridgesHint telling a caller it is the source of truth and apps.*.bridgeRunning is the legacy view of the same data.

Verified live just now on a Windows target: status.desktop.apps.{kicad,fusion360,browser}.bridgeRunning all true while bridge_list shows all three running with instanceCount:1. They agree. An AI trusting status no longer concludes a running bridge is dead.

Part 2: alive-but-never-binds → raw reqwest string, no recovery. FIXED (recover_not_listening).

The dedicated kicad + puppeteer request paths now catch a CONNECT-class send failure and route it through recover_not_listening instead of surfacing the raw string. It:

  1. Reaps whatever holds the port reservation (a process that came up but never bound), so the NEXT verb spawns a clean instance that health-waits for the listener.
  2. Returns a rich payload: errorCode: "bridge_not_listening", killedPids, the bridge's logPath, processWasLive, and a _hint explaining this is a startup hang (process up, never reached its listen call), telling the AI to RETRY and where to read the bridge log.

It is deliberately ONE reap + one clear error, not a retry loop. Your own report flagged that "bridge_kill + auto-respawn did NOT recover it" and that repeatedly respawning a bridge that wedges on startup is the bludgeon to avoid, so the design is: break the wedge once, hand back an actionable error, let the caller's retry do the clean respawn. So an AI now gets bridge_not_listening with a fix, not a naked transport error, and does not need the AD process restart that was your only recovery before.

Part 3 (the trigger): puppeteer's.

You called it a routing note, and it is: the process coming up but never listening because startup session-recovery blocks on a stale Chrome profile lock is pup's to fix (bind the listener BEFORE the recovery work). recover_not_listening's hint says exactly that and points at the bridge's own wiki page. AD's job was to stop turning that into an unrecoverable wedge with an opaque error, which is what these two fixes do.

Thanks for the response-signature table and the minute-by-minute timeline. The "status disagrees with bridge_list" contradiction is what pinned part 1 unambiguously.

Log in to reply.