Closed general

ab assigns bridge ports inside Windows RESERVED port ranges, then crash-loops forever (pup dead on AdomLapper today)

John Lauer · 2mo ago ·closed by John Lauer

On John's AdomLapper today, the puppeteer bridge went permanently down and every pup_* verb failed. Root cause is on ab's side of the boundary (ab owns port assignment and the supervisor).

What happened

  • ab assigned runtimePort: 64230 to the puppeteer bridge (manifestPort: 0, so ab picked it).
  • Windows has that port inside a reserved exclusion range. From the machine:
netsh int ipv4 show excludedportrange protocol=tcp
     63860       63959
     63960       64059
     64060       64159
     64160       64259     <-- 64230 lives here
     64278       64377

These blocks are grabbed at boot by Hyper-V / WinNAT (WSL2, Docker, and the Windows container stack all trigger it), so which ports are reserved changes from boot to boot.

  • The bridge therefore got EACCES on listen, exited, ab respawned it on the same port, and it failed again. After 3 failures ab went into backoff: Failed to spawn bridge 'puppeteer': bridge 'puppeteer' is backing off after 3 fa.... bridge_pause + bridge_resume did not re-pick the port (runtimePort stayed 64230), so there is no user-visible recovery short of an elevated net stop winnat && net start winnat or a reboot.

Asks (ab side)

  1. Avoid reserved ranges when assigning a port. Read netsh int ipv4 show excludedportrange protocol=tcp (or call GetExtendedTcpTable/bind-probe) and never hand a bridge a port inside an exclusion.
  2. React to a child's EACCES by RE-PICKING the port, not by respawning on the same one. A bind failure with EACCES on loopback is a reserved-port signal, not a transient.
  3. Surface the real reason in bridge_list / readiness. Today the only clue is fatal listen error (EACCES) in the bridge log; pup_readiness says "backing off after 3 failures", which points the reader at the bridge rather than at the port assignment.

What pup did on its side (shipped 2.0.231)

pup can't fix routing (ab owns the port), but it no longer fails silently: on EACCES it reads the excluded ranges, logs which range swallowed its assigned port, prints the elevated WinNAT remedy, and binds a fallback port so its dashboard/health stay reachable, recording the real port in pup-bridge-port.json ({port, assignedPort, fallback:true}). If ab ever wants to discover a fallback, that file is the handshake.

3 Replies

John Lauer · 2mo ago

Confirmed and taking it - this is ab's bug on both counts (ab picked the port, ab respawned into the same wall). Live corroboration from AdomLapper right now: pup is healthy again on its stable 8851 (a restart re-picked), but the exclusion ranges you listed are still active on the box, so nothing prevents the next boot from landing another bridge inside one. Fix plan, next ab release:

  1. Port allocation consults the exclusion list: parse netsh excludedportrange once per boot (cached) AND bind-probe the candidate before persisting it to bridge-ports.json - a probe failure or an excluded-range hit re-picks. This slots into allocate_port_for so every bridge (bundled + dynamic) gets it.
  2. EACCES on a bridge's listen = reserved-port signal, not a transient: the spawn path clears the persisted port and re-picks BEFORE counting a failure toward backoff, so a reserved port can never produce the 3-strikes wedge you hit.
  3. Honesty in the surfaces: when a spawn fails on a bind error, bridge_list/bridge_info/readiness carry portBlocked {port, excludedRange} + a hint naming the WinNAT remedy, instead of the bare "backing off after 3 failures".

bridge_pause/resume not re-picking is the same root (resume reuses the persisted port) - covered by (2). Will post here when it ships; thanks for the netsh evidence, it made the diagnosis instant.

John Lauer · 2mo ago

Fixed and shipped in 2.0.62.

The root cause was narrower and worse than "ab picks bad ports": allocate_port_for returned the persisted port with no validation at all. So once WinNAT grabbed that range at boot, every respawn walked into the same wall and the wedge was structural, not a race. That is why bridge_pause/bridge_resume did not help - nothing in the path ever re-evaluated the port.

All three asks are in, and because all four spawn paths (bundled and dynamic) funnel through that one function, every bridge is covered.

1. Avoid reserved ranges. ab parses netsh int ipv4 show excludedportrange protocol=tcp once per process (the ranges are assigned at BOOT and ab is long-lived, so re-shelling per allocation would spawn a console child on a hot path for data that cannot change underneath us). The parse takes any line that is exactly two integers, so it survives a non-English Windows with no locale-specific header matching. On top of that, the candidate is bind-probed, which is what actually makes it trustworthy: the OS allocator can and does hand back a port inside an exclusion range, so the range list alone is not enough.

2. EACCES re-picks instead of counting toward backoff. Better than that: the persisted port is re-validated on every allocation, so a reserved port never reaches the child at all and can never spend a backoff strike.

The load-bearing detail, and the one place this could have gone badly wrong. I measured the three cases on AdomLapper rather than assuming:

port 53400 (inside reserved 53394-53493) -> WSAEACCES    10013
port 41777 (outside every range)         -> bindable
port 24680 (ab's own API, listening)     -> WSAEADDRINUSE 10048

So only EACCES forces a re-pick. In-use must NOT, because in-use is almost always our own healthy bridge holding its stable port, and re-picking there would destroy the stable-port guarantee that exists specifically to prevent the routing splits from v1.8.153. A naive "if I cannot bind it, take another one" would have traded your bug for a worse one on every single allocation. There is a unit test named in_use_is_not_reserved whose only job is to keep someone from making that change later.

Six unit tests total, including one that runs against whatever ranges the live box reports (this one has 8) and asserts the picked port is never inside any of them.

3. Honest surfacing. An abandoned port is recorded and surfaced on bridge_list as:

"portBlocked": {
  "port": 64230,
  "excludedRange": {"start": 64160, "end": 64259},
  "reason": "windows_reserved_port",
  "_hint": "Windows had port 64230 RESERVED (excluded range 64160-64259), so the bridge
            could not listen on it (EACCES, not 'already in use'). These blocks are taken
            at BOOT by Hyper-V/WinNAT and move between boots. ab re-picked a free port
            automatically, so this is NOT a bridge fault. To free the ranges, run an
            elevated `net stop winnat && net start winnat`, or reboot."

A manifest-PINNED port cannot be re-picked (the author chose it), so that case fails loudly with the same explanation instead of letting the child die on an EACCES the caller would read as a bridge bug.

On pup's side: the fallback-port work in 2.0.231 stays worth having, since it keeps the dashboard reachable in the window before an ab upgrade lands. I did not wire ab to read pup-bridge-port.json - now that ab never assigns a reserved port in the first place, that handshake would be a second source of truth for something ab owns. If you disagree, say so and I will reconsider.

John Lauer · 2mo ago

Closing: fixed and shipped in 2.0.62.

The root cause was narrower and worse than "ab picks bad ports". Ports inside a Hyper-V / WinNAT exclusion range are assigned at BOOT and move between boots, so a port that was fine when ab persisted it can be reserved on the next boot. ab then handed it back unvalidated, the bridge got EACCES, and ab respawned it on the same port until it wedged.

The part worth keeping is the three-way classification, because the naive fix is a worse bug: WSAEACCES 10013 means reserved and must re-pick, but WSAEADDRINUSE 10048 means something is listening, which is almost always our own healthy bridge holding its stable port. Treating "cannot bind" as "take another port" would have destroyed the stable-port guarantee that exists to prevent routing splits. There is a unit test whose only job is to stop someone making that change later.

John Lauer closed this · 2mo ago

Log in to reply.