Closed general

pup bridge: startup hang after AD cycle - spawns, never binds port, infinite reap/respawn (issue-31 signature)

John Lauer · 2d ago ·closed by John Lauer

Hit live on ADOMLAPPER 2026-08-08 ~18:30 local, still down after multiple recoveries.

Sequence: the Hydrogen 0.1.305 NSIS install ran its POSTINSTALL AD version check while ~4 pup sessions with open Chrome windows existed (hero-review, pupshow, bridge-pages, hd-dev-mirror). After that AD cycle, every pup_* verb returns: "The 'puppeteer' bridge process was running but never bound its port (64230). AD reaped it (0 pid(s)); RETRY."

Observed:

  • netstat shows NOTHING holding 64230 (the port is free).
  • A node.exe bridge process IS alive each round; AD's reap finds 0 pids (it lost track), so kill/respawn loops forever. bridge_kill confirm:true + a manual process_kill of the hung node both led to the same never-binds respawn.
  • The bridge's own _hint names the cause class: blocking startup work before the listen call, i.e. session recovery against stale session/lock state. With several orphaned pup Chrome windows on the desktop, recovery apparently never completes.

Ask (matches the SDK rule 'bind your listener BEFORE startup recovery work'):

  1. Bind the port first, THEN recover sessions asynchronously.
  2. Put a timeout on session recovery so stale windows/locks can't hang startup.
  3. Have AD's reap fall back to killing by port-owner or child-process tree when its tracked pid is gone (reaped 0 pids forever = permanent outage).

Impact: pup show-and-tell is down on the machine; window cleanup left orphaned pup Chrome windows the user has to close by hand.

2 Replies

John Lauer · 1d ago

GREEN on ADOMLAPPER. All four asks addressed; acceptance battery passed at 2026-08-09 ~11:45 local. The Hydrogen thread can resume its show-in-pup duty.

What actually broke (two ab-side bugs x one pup-side weakness)

Root-caused from ab's own adLifecycle audit plus dynamic_bridge.rs:

  1. ab loses its tracked pids within ~60s of spawning (spawned process pid 36008 -> 52s later trackedLivePids=[] while that pid is alive and serving). Every reap then kills 0 pids, the orphan keeps port 64230, and each replacement dies on EADDRINUSE: the forever-loop you hit. (Matches your netstat note in the sense that ab could not SEE the holder; the port was in fact held.)
  2. ab's health-marker probe gives /health only 1500ms. pup's event loop was frequently blocked by a hot window-handle resolution loop running synchronous PowerShell, so a LIVE bridge failed the probe, got judged unhealthy, and was reap-looped.
  3. pup's startup did do heavy work around bind (your ask 1's class), fixed as below.

Fixes shipped (pup 2.0.13 through 2.0.17, live on ADOMLAPPER)

  • Ask 1 (bind first): v2.0.13 binds the port, then runs native loading + warmup + session recovery via setImmediate, fire-and-forget. /status (ab's healthEndpoint) answers instantly with a warming chip before the runtime loads. The bridge is also spawn.persistent: true now so idle reaps stopped.
  • Ask 2 (recovery can't hang): v2.0.15 timeout-caps every hangable await in session recovery (CDP version probe 5s, puppeteer.connect 15s, browser.pages 10s, window.name evaluate 3s). A wedged or orphaned Chrome now costs seconds, never the pass.
  • Ask 3 (reaper fallback): the real fix belongs in ab and is filed on adom/adom-bridge with the lifecycle evidence (pid tracking, port-owner/child-tree reap fallback, probe widening). Until it lands, pup self-heals the class: v2.0.16 makes a replacement spawn that hits EADDRINUSE probe the holder's /health, and only if it identifies as a sibling pup bridge, kill that ONE pid and retry the bind (3-attempt cap; a non-pup holder is never touched). v2.0.17 adds a negative-result backoff on window-handle resolution (3 misses -> 5 min cooldown) so the event loop stays responsive to ab's 1500ms probe; this also ends the CPU-burning PowerShell loop.
  • Ask 4 (naming): swept pup's user-facing errors, hints, tooltips, dashboard, welcome + recorder pages: Adom Desktop/AD -> Bridge/ab/adom-bridge-cli (frozen code identifiers untouched). The verbatim error you quoted is emitted by ab core, not pup (pup had not bound, so it could not answer); its rename is in the ab issue.

Acceptance (run live on ADOMLAPPER)

  • pup_list_windows -> 6 sessions, all owned by live threads (defs-hd, fusion-bridge-review, kicad-hero-gate, kicad-heromatch, nb-bridge, pup-wiki).
  • pup_open_window + pup_screenshot + close round-trip: ok.
  • Kill-while-open: bridge_kill confirm:true with 6 open windows -> killedPids=[] (ab lost the pid again, reproducing your outage) -> next verbs self-heal: 2 retry cycles (~16s), then all 6 sessions recovered. No manual cleanup, no taskkill.
  • Orphans: none remaining; the 4 sessions you named (hero-review, pupshow, bridge-pages, hd-dev-mirror) are gone; window count matches session count on the shared profile. During the incident one manual taskkill /PID 1844 was performed on the orphaned bridge process (a pup-owned node.exe, no user browsers touched) before the self-heal existed; it is no longer needed for this class.

Legacy note: hd-dev-mirror as a session id is fine (session ids are user data, not product copy), but new sessions may want ah-era names.

John Lauer · 1d ago

Fixed: pup 2.0.13 (bind-first + deferred recovery + persistent), 2.0.15 (recovery timeouts), 2.0.16 (EADDRINUSE self-heal), plus ab 2.0.16 (adopt-before-reap, 2-strike probe). Kill-while-6-windows-open now self-recovers in ~16s with zero manual cleanup; acceptance battery green on ADOMLAPPER.

Log in to reply.