Closed bug report

A healthy persistent bridge is terminated ~17 s after every spawn with nothing in its ad.log (ab 2.1.168, altium-bridge#30)

Colby Knox · 4h ago ·closed by Colby Knox

ab 2.1.168 · altium-bridge 0.2.94 (Rust, spawn.persistent: true) · Windows 11, Spanish locale · Altium Designer 26.11.1 · reporter's machine in altium-bridge#30

What happens. On one user's machine the altium bridge is ended about 17 s after every spawn while it is healthy. ab starts a new one about 94 s after the previous spawn (σ 4.6 s over 54 cycles), and the cycle repeats with no calls at all. altium.ad.log shows only spawned process pid … lines: no REAPED, no EARLY-EXIT, no bridge_kill. Every agent verb that takes longer than the remaining seconds dies with ConnectionResetError. The user has seen 500+ respawns since Nov 5.

What the bridge's own logs prove (0.2.93/0.2.94 were built to answer exactly this):

  • It answers its own port at 0–1 ms right up to the end. A self-check thread connects to 127.0.0.1:<port>/ping every 5 s; the last tick is at 15 s, sometimes 20 s, and then nothing.
  • It does not exit by itself: no panic, and no Rust fatal message (stderr is redirected to a file, and the redirect is proven working on that machine). No "listener stopped accepting" line, and the bridge has no exit timer.
  • Windows records no Application 1000/1001/1026 for it, and Microsoft Defender shows no detection or ASR event.
  • So the process is terminated from outside, roughly 17 s after spawn, and the parent is ab.

Quiet run, no traffic (from the user's comment on altium-bridge#30):

09:34:32  pid 31508 started   ticks 5 s, 10 s, 15 s answering in 0-1 ms, then gone
09:36:21  pid 31324 started   ticks 5 s, 10 s, 15 s answering, then gone
09:38:02  pid 15380 started   agent_start in flight; ticks to 20 s, then gone
ad.log:   [ab 2026-11-10 09:34:32] spawned process pid 31508 (altium-bridge.exe)
          [ab 2026-11-10 09:36:21] spawned process pid 31324 (altium-bridge.exe)
          (no other lines)

The same bridge build does not do this elsewhere:

  • Colby-ROG2, public channel, not dev-pinned: one 0.2.94 process has been up 22.5 h.
  • DESKTOP-BAR6S0F, ab 2.1.188: processes run for hours. A test build whose /health took 30 s on purpose was also left alone there, while ab probed it every ~6.5 s.

Possibly relevant. The bridge's /health handler calls back into ab: process_list, then desktop_enum_windows, cached for 2 s. If ab 2.1.168 health-probes a fresh bridge and that probe waits on ab itself, the probe could miss. #258's third answer describes ab replacing a live bridge after a missed health probe, fixed in 2.1.188 together with "logs why before it ever replaces one". That fits this machine, which is still on 2.1.168. Whether the fix also covers this cycle (no calls in flight, replacement ~17 s after spawn, next spawn ~77 s later), and whether 2.1.168 replaced bridges without logging it, only ab's code can say.

Asks

  1. In 2.1.168, which paths can terminate a spawned bridge without writing to <name>.ad.log? Is a spawn-time or sweep health probe one of them, and what timeout does it use?
  2. Does 2.1.188 cover this case (a missed probe on an idle bridge), and is 2.1.188 on the public channel yet? The user would update to it as the next test.
  3. Every termination ab makes should name itself in <name>.ad.log, with its reason and exit code, so a silent kill can't look like a crash.

The altium bridge will also stop calling ab from /health, so that its probe answer never depends on ab. That is a mitigation on our side, not an answer to why the process is killed with nothing logged.

3 Replies

Colby Knox marked this in progress · 4h ago
Colby Knox · 4h ago

Answers from ab's code (2.1.168 = d174e1fe, 2.1.188 = 6d85a3d2).

1. Which paths in 2.1.168 end a bridge without writing to <name>.ad.log

The only automatic one is the persistent supervisor (dynamic_bridge.rs ~2136-2155):

  • When it runs: on every health tick (lib.rs:6871, a 15 s sleep plus that tick's work).
  • What it does: it probes /health. If that misses, it waits 1 s and probes again. If both miss, it calls ensure_bridge_running, which runs reap_bridge and reap_port_orphan (taskkill /F) and then spawns again.
  • What counts as a miss: any probe slower than HEALTH_PROBE_TIMEOUT_SECS = 6 s, so a slow /health is treated as a dead bridge.
  • What gets logged: only ab's main log (supervisor: 'altium' missed 2 probes, respawning), never ad.log.

The other paths that log nothing are all started by a person or a call, not automatic:

  • restart_bridge (verb and GUI button) and the GUI bridge_kill
  • kill_persistent_bridges (tray Quit, shutdown, bridge_stop_all)
  • the reap inside bridge_install

Duplicate collapse, update reaps and the bridge_kill verb do log. None of them records an exit code.

The spawn-time readiness poll never kills a bridge. It accepts a slow /health for up to about 6.5 min (60 polls of 0.5 + 6 s), which is why a slow test build looks fine for the first few minutes.

How that fits altium-bridge#30:

  • The cause: altium's /health calls back into ab: process_list, which runs PowerShell Get-CimInstance with no timeout, then desktop_enum_windows. Ticks are at least 15 s apart, so the 2 s cache is always stale, and every probe pays for a fresh callback. A timed-out probe doesn't cancel that work, so slow callbacks pile up.
  • The timing: once a few probes in a row run past 6 s, the supervisor kills the bridge and spawns the next one in the same call. So the old process dies roughly a second before each spawned process pid line, about 94 s after its own spawn. Its server stops answering at ~17 s, the first supervisor probe after the spawn. The ~94 s period is several of those ~25-30 s ticks, and it can't be pinned down further from the code alone.
  • How to confirm it on that machine: look in ab's main log (identifier inc.adom.bridge under %LOCALAPPDATA%) for supervisor: 'altium' missed 2 probes right before each spawning 'altium'. Or run tasklist /FI "PID eq <pid>" at spawn+30 s and again at spawn+80 s: alive, then gone, means it's this path.

2. Does 2.1.188 cover it?

No, not for an idle bridge:

  • The #258 change only spares a bridge with calls in flight, and the supervisor has none. The 6 s probe and the double probe are unchanged.
  • What 2.1.188 does add is a line in ad.log just before that reap: health probe at <url> did not answer (TimedOut(...)); stopping pids [...]. On this machine, that line would name the cause.
  • 2.1.188 is on insiders only. Public is still 2.1.168.

3. Logging every termination

Agreed. Still missing in 2.1.188:

  • the port-orphan reap when ab has no pid on record;
  • restart_bridge (verb and GUI) and the GUI bridge_kill;
  • kill_persistent_bridges;
  • the bridge_install reap.

No path records the exit code yet, and the EARLY-EXIT watcher only watches the first 12 s.

On the altium side, taking ab out of /health is the right fix. A health answer that never waits on ab can't time out this way.

Colby Knox · 3h ago

Fixed in Adom Bridge 2.1.189, which is on the insiders channel now. Public is still 2.1.168.

What changed

  • A slow /health no longer gets a bridge replaced. A persistent bridge that takes connections but answers slower than the 6 s limit stays up. After 4 slow supervisor ticks in a row ab asks once more with a 30 s limit. A bridge that answers that is slow, not stuck, and stays up. Only one that leaves it unanswered is replaced. A port that refuses connections is still restarted at once.
  • A verb that arrives while /health is slow goes through to the bridge. Before, the call path replaced the bridge first and the caller waited for the respawn. This applies to routed verbs, prewarm, the stable URL route and the verb runner.
  • process_list has a 20 s limit on its PowerShell query, and answers process_query_failed when the query does not finish, not an empty list. The 30 s probe is longer than that limit on purpose, so a health handler waiting on process_list still gets to answer.
  • Every stop writes its reason to <name>.ad.log: the supervisor, the port-orphan reap, restart_bridge, the Restart and Kill buttons, bridge_stop_all, tray Quit, shutdown and bridge_install. The early-exit watcher now covers the first 30 s after a spawn (was 12 s).
  • status.bridgeHealth and the SDK runtime skill describe the rules, including: never call an ab verb from a health handler.

Proof on our test machine (2.1.189, a test bridge with a settable /health delay)

Case Result
/health takes 8 s, bridge left alone Same pid after 9.4 minutes. 2.1.188 would have replaced it at about 7 minutes.
A verb sent while /health takes 8 s Answered by the same pid in 12 s. No restart.
/health hangs (600 s) Replaced after 8 slow ticks plus an unanswered 30 s probe, with the reason in ad.log.
restart_bridge Logged in ad.log with who asked.

The ad.log from the first and third cases:

11:50:38 spawned process pid 14516 (python.exe)
11:57:50 supervisor: health probe ... did not answer within 6 s (TimedOut(6008)). The bridge still takes connections, so it stays up; ...
12:00:02 supervisor: health ... has been slower than 6 s for 4 ticks in a row, but it answered a 30 s probe: slow, not stuck, so the bridge stays up.
12:00:27 health probe ... did not answer within 6 s (TimedOut(6002)), but the bridge takes connections, so it stays up and the caller's verb goes to it
12:03:18 supervisor: replacing the bridge: health probe went unanswered on 8 supervisor ticks in a row and then for 30 s more (last: TimedOut(30004))
12:03:30 health probe ... did not answer (TimedOut(6007)); stopping pids [14516] and starting a fresh instance
12:03:30 spawned process pid 9880 (python.exe)

Answers to the three asks, updated

  1. The silent paths in 2.1.168 are listed in the earlier comment. All of them log in 2.1.189.
  2. 2.1.188 did not cover an idle bridge. 2.1.189 does.
  3. Done, with one gap: the exit code is recorded only where ab holds the process handle (bridge_stop_all, tray Quit, shutdown). The other paths name who stopped the bridge and why, without an exit code.

Known and not in 2.1.189

  • The restart_bridge, Restart button, Kill button and bridge_uninstall lines print pids [] even though the process was stopped. The fix is written and unit-tested for the next release.
  • A verb sent to a bridge with a slow /health waits about 12 s for two probes before it is forwarded.

I'm leaving this open until 2.1.189 reaches the public channel, since the reporting machine follows public.

Colby Knox · 3h ago

2.1.189 is on the public channel now (promoted 2026-11-10 18:15 UTC), so machines on public get the fix with their next update. The reporter on altium-bridge#30 has been told.

Before the promote, the fleet sweep ran against the test machine on 2.1.189: 37 of 37 applicable checks passed (status, updater, bridges, shell, files, input, capture, recording, toast), and the loopback checks passed on the machine itself.

Closing. The two known gaps from the previous comment (the pids [] wording on the restart, Kill and uninstall lines, and the 12 s wait before a verb is forwarded to a slow bridge) ride the next release.

Colby Knox closed this · 3h ago

Log in to reply.