Altium - the Altium Bridge

Root cause found, with hard evidence. It is neither "the loop was dead so it must be the servers" nor purely the loop: the decisive fact is that the key piece of evidence in this issue (agentAlive: false during the freezes) was a false negative. The bridge has a workspace split-brain, and every observer read the wrong workspace.

What actually happened (today's last freeze, from ab's log + files pulled from the box)

1. 17:49:36, the staging wiki page (web:https://helium.stg.adom.internal, direct_api) fired altium_install_library_bundle. Altium was closed, so 0.2.8's auto-launch kicked in.
2. Verb dispatch resolves the target instance ONCE, at the start of the verb. Altium was not running yet, so resolution returned not_running and the verb proceeded with NO instance selected. The verb then launched Altium itself and never re-resolved.
3. The agent bootstrapped on the first agent call with no instance key, so workspace_dir() fell back to the LEGACY root (bridges-cache\altium\agent\ instead of agent\ADDevelop). Proven: I pulled the rendered AdomAgent_9ef974bc70.pas from the box; its baked WORKSPACE is the legacy root, and agent.json + hb.txt sat there while agent\ADDevelop\ was empty.
4. From that moment every status and teardown verb looked at agent\ADDevelop\ and saw nothing: altium_status said agentAlive false while a live loop heartbeated in the legacy root (I watched hb.txt tick at ~108 ms cadence, requests=4, gen matching the current template digest).
5. 17:51:43 the e2e thread called altium_close ("close cleanly via the verb"). misc::close gates its agent teardown on agent::is_alive(), which reads the per-instance workspace: false, so lifecycle::stop() WAS SKIPPED and the polite WM_CLOSE went out with the agent resident. The close wedged (retry at 17:52:32 with waitSec 35 also failed).
6. The user eventually pressed X on the still-open window, the desktop hard-froze, and at 17:56:40 the thread force-killed X2 (seat stranded again). The loop's heartbeat ran right up to the kill (frozen at the kill tick, agent.json has startedTick but no stoppedTick: died with the process, never exited).

So the "agent was already dead" premise is wrong: it was alive the whole time, in a workspace nothing else reads. That is also why the polite WM_CLOSE failed before the user ever pressed X, and why the 0.2.8 watchdog logged nothing.

Reconciling the two theories

• The resident loop wedging Altium's shutdown is the evidence-backed proximate cause of the 17:5x freeze (close verb skipped the teardown, loop provably resident through the close).
• The earlier 15:10 freeze was the other flavor: the agent was in the RIGHT workspace and its heartbeat died at close time. While a script runs, the only message pump is the agent's own Application.ProcessMessages, so a user X-press always dispatches inside the agent's stack frame and the shutdown can park above the live script. Note: the 0.2.8 template already checks Client.IsQuitting each tick and it demonstrably never helped, so that signal cannot be trusted either.
• The PCB/SCH scripting servers (ServersUp at every agent start) do stay loaded after the loop stops, and this issue's ask stands: stop what we started. But on the evidence they were not the trigger for the freezes we can reconstruct; the resident script was.

Fix plan (0.2.9, dedicated release)

Bridge (bridge-rs):

1. Kill the split-brain class: agent bootstrap REFUSES to run without a resolved instance key (no more silent legacy fallback), and install_library_bundle re-resolves the instance after its own launch before any agent call.
2. Teardown and status become workspace-union: altium_close and lifecycle::stop stop EVERY workspace with a live heartbeat (legacy included), and altium_status/agent_status report a loop in ANY workspace, so status can never again say "not resident" while the engine is occupied.
3. Watchdog stays as backstop, but its detection must not depend on ab window enumeration, which hangs exactly during a freeze (measured in both incidents).

Agent template (the dedicated-release, live-close-test part): 4. Restructure the resident agent from a blocking loop to a timer-driven tick (hidden script form + TTimer): between ticks NO script is executing, so a user close finds the engine idle and has nothing to wedge on. This removes the ProcessMessages re-entrancy class entirely instead of racing it. Engine support for this is being probed live before any template change ships. 5. On agent exit, Client.StopServer for servers this agent started (and only those).

Acceptance test, exactly as specified here: auto-launch install, altium_state clean, then a real X press with the desktop staying responsive and X2 exiting clean, three times, coordinated with Colby before each run.

Two side findings, filed for completeness: the pushed bridge-rs does not compile (verbs.rs references board::SchematicData and board::cross_check, missing from the pushed board.rs; reconstructing so the release lane works), and the staging page at helium.stg reached this machine's bridge via direct_api all day, so the one-click gate needs to cover staging as well as prod or the test page keeps arming the trap.

Probe campaign complete (live on Colby-ROG2, ~10 throwaway scripts, every modal auto-reaped or hand-cleared). Results below are paid-for engine facts; recording them so nobody re-buys them.

What the DelphiScript engine CANNOT do (all measured today)

1. Nothing survives script return. A dfm-created TTimer armed by the entry procedure never fires once after the procedure returns (form and handlers die with the script). A timer-resident agent is impossible.
2. The script's VCL context is a sandbox. With Altium fully running: Application.MainForm = Nil, Screen.FormCount = 0. There is no reachable form to hook OnCloseQuery on.
3. Runtime event assignment is illegal in every spelling. X.OnEvent := Handler and := @Handler are compile errors; := Nil compiles and then halts at runtime. No OnMessage lane.
4. TApplicationEvents is not in the dfm class registry ("Error Reading Form: Class TApplicationEvents not found"), so declarative wiring of application-level events is out too.
5. Client.StopServer exists but rejects both plausible one-argument forms (name string, module interface) with a "Wrong number of params" debugger prompt, and the attempt parks the loop behind that prompt, which is the same hazard class as the shutdown wedge. Server hygiene is not safely implementable from the script. I recommend dropping that half of ask 1; whether the servers matter at all gets decided empirically by the acceptance test once the agent-residency fixes land.
6. Application.Terminated IS readable (false at idle). It goes in as an extra in-loop exit signal beside Client.IsQuitting (which is proven to never fire in time).

Conclusion: no interception design is possible in this engine. The complete fix cannot be "the agent notices the close before Altium does"; it has to be "there is (almost) never a resident script for the close to wedge on, and when there is, something that works DURING a freeze ends it fast."

Why the 0.2.8 watchdog missed (root-caused in code)

Its detection runs entirely through ab (process_list + desktop_enum_windows), and during a freeze those calls hang or return empty; an empty process list makes it conclude "no X2" and drop nothing. Detection has to be native and non-messaging (EnumWindows/GetClassNameW/GetWindowThreadProcessId; never GetWindowText, which sends WM_GETTEXT into hung windows).

Final 0.2.9 design

Bridge (Rust):

• Workspace-union teardown and status (done, tested): altium_close stops every live workspace's loop before WM_CLOSE; status reports agentResidentAnywhere + liveWorkspaces so a loop can never hide again.
• install_library_bundle re-resolves the instance after its own auto-launch (kills the legacy-workspace split-brain at its source) and STOPS the agent when the install lane finishes, so the one-click leaves no resident script at all. A user who closes Altium after an install finds a clean engine.
• Watchdog v2: native, non-messaging window/process detection; on "main window gone + X2 alive" it drops stop files into every live workspace within 2 s (a pumping loop consumes one within ~100 ms; today's 17:5x loop pumped for minutes and would have exited immediately). Escalation: if the process still will not exit after a hard threshold with the desktop at risk, announce via toast and kill it, recording the seat consequence. A ten-second self-healing blip instead of a bricked desktop.
• Dialog catalog additions from today: "Error Reading Form" (Ignore/Cancel/Ignore All: evaded both the reaper and check_dialogs) and the script debugger's "Continue execution?" prompt (title "X2", empty body: must be REPORTED by check_dialogs, never auto-answered).

Template (small, all proven-safe):

• Application.Terminated added to the loop's exit checks.
• Idle auto-stop shortened from 30 min to 5 min: the freeze window shrinks to "a verb is in flight or ran in the last five minutes".

The acceptance test stays exactly as specified in this issue, and runs before the release ships.

0.2.9 is implemented and built (artifact adom-bridge-altium-v0.2.9.zip, sha 827028a68fcb, deliberately unpublished so no box auto-updates before the live acceptance). Everything from this issue's fix plan is in, plus the probe-campaign consequences; the changelog entry carries the full list. Highlights against this issue's asks:

• Bootstrap can no longer land in the legacy workspace (resolve-retry then refusal); close/status/state are workspace-union.
• Watchdog v2 detects natively (non-messaging Win32; "cannot answer" is not "none") and carries the zombie-breaker: toast then terminate after 45s of window-gone/X2-alive with our loop involved.
• The install lane stops the agent when it finishes, so the one-click leaves an idle engine; template idle-stop is 5 minutes; Application.Terminated joins the exit checks.
• Server hygiene (ask 1's StopServer half) is deliberately dropped: both plausible signatures fail with debugger prompts, per the probe campaign above.

Next: manual deploy to Colby-ROG2 (files into bridges-cache + bridge_kill, no manifest publish), pre-flight probes (close_document identifier, panel select, bootstrap refusal), then the acceptance: auto-launch install, state clean, three warned X-presses with the desktop staying responsive. Publish to the manifest only after that passes.

User-verified acceptance on Colby-ROG2, 0.2.9 (manual deploy): Colby ran the REAL one-click Export from the staging wiki page, the install completed, and he closed Altium with the window X himself. His words: "it worked beautifully and i was able to x out cleanly." No freeze, no wedge, no seat strand. That is the exact acceptance this issue defined, performed by the user who hit the original brick five times. Leaving the close to you (your release, and the manifest/fleet publish is still pending), but from the wiki side this is fixed.

Closing on the strongest possible evidence: 0.2.9 published (manifest live, sha 739a5b378564) and the acceptance was performed by the user who hit the original brick five times, with the real gesture on the real lane: Colby ran the one-click Export from the staging page and closed Altium with the window X himself. "it worked beautifully and i was able to x out cleanly." No freeze, no wedge, no seat strand.

What made it safe, in one paragraph for the record: the workspace split-brain is closed (bootstrap refuses to run unresolved; teardown and status are workspace-union), the install lane stops its agent when it finishes so a later X-press finds an idle engine, the close verb runs the REAL exit process (Client:QuitFromEDAClient - the skinned frame ignores WM_CLOSE/SC_CLOSE), idle residency is capped at 5 minutes, and the reworked watchdog detects the true freeze signature (window PRESENT but hung, via non-messaging IsHungAppWindow) with a 45s zombie-breaker scoped to bridge-involved closes. The full probe record and the engine facts live in this thread; the acceptance lever (altium_close unsafeSkipTeardown) stays in the verb for future freeze drills.

Residual risk, stated honestly: the watchdog's recovery under a REAL wedge has not been exercised (every close since the fix has been clean, which is the point). Its stage-1 stop-drop is live-proven, its kill mechanics are the same native call used in this session's emergency, and it logs and toasts when it acts, so the first real-world firing will be visible. A bounded freeze-drill design is on file if we ever want to exercise it deliberately.