Adom Hydrogen (platform base)
Public Made by Adomby adom
Layer Hydrogen's workspace onto a standard Adom workstation.
Hydrogen Runtime Recovery: bounded stop-loop diagnosis, evidence and safe resume #4
Linked full evidence
Closes/relates to Hydrogen Bootstrap issue #10, which contains the sanitized full accessible history matrix, test outputs, exact patch and known limitations.
Addition
Adds optional hydrogen-runtime-recovery skill and companion Python tools:
- read-only, deduplicated no-progress/loop/missing-app/auth/service triage;
- Hydrogen Doctor integration and dependency classification;
- guarded same-work-item recovery with native revision hashes, exclusive lock, time/retry budget and safe rollback;
- independent verified Claude background-session ownership handoff;
- opt-in bounded dispatcher (dry run/shadow mode default) and private event evidence;
- orchestration handoff to Contribution Assurance only when a software defect is reproducible.
Hydrogen Bootstrap now installs the new skill's companion tools/*.py and offline tests/*.py, without changing existing skills' install layout and without enabling background monitoring or auto-repair on installation.
Generic installer correction
In the current install.sh, the final [ "$purged" -gt 0 ] && log ... expression can terminate an otherwise successful clean install under set -e when there are no retired hd-* skills. Replace it with an explicit if so zero purges are successful. Reproduced and corrected with a clean, temporary-HOME sandbox.
Evidence and acceptance
- Base:
70f04086e01782a55f505809862f042494e8eea5(v0.5.83). - 116/116 isolated simulation and regression tests passed.
- Independent clean-HOME installer test delivered all skill/tool/test files; installed test suite also passes.
- ADOM package lint passed with zero errors and existing nonblocking metadata/import warnings.
- All 199 adjacent commit comparisons in 200 returned revisions verified. ADOM exposed no earlier page history through its capped CLI; the earliest observed revision is marked unknown, not falsely described as the origin. Complete captured matrix is attached to Issue #10.
- No user credentials, private CAD, external telemetry or user file paths are included.
Deployment boundaries
POSIX process inspection/locking tested in Linux/WSL; macOS and native Windows acceptance remain maintainer work. New skill install alone does not enable a user-level timer, Stop hook, agent dispatcher or remote publishing. Native AI Flow, safety gates, manual auth and explicit upstream submission approval stay authoritative.
This PR changes only the proposed branch. It does not publish a package, merge upstream or replace any running Hydrogen binaries.
Diff Skip to comments (1)
@@ -1,91 +1,106 @@⋯ 88 unchanged lines ⋯ sudo-free (`needs_sudo: false`; the retired updater daemon was the only sudo dependency). Build the golden image on the laptop via Adom Bridge or in CI, never in the cloud container.+++## Hydrogen Runtime Recovery (opt-in)+The `hydrogen-runtime-recovery` skill detects stopped/idle Hydrogen work,+missing required executables, services and configuration, and repeated+unchanged tool errors. Bundled Python standard-library tools implement a+read-only classifier, recovery gate, bounded safe dispatcher and independently+checked background-session handoff.++**Install does not enable monitoring or automatic repair.** An owner must+create per-run bindings and explicitly opt into an observer. Linux/WSL POSIX+process and lock interfaces are required for the current execution controls.+Never publish issue/PR data without the owner's distinct consent; this+contribution contains no customer CAD files or private runtime logs.+@@ -1,54 +1,62 @@⋯ 3 unchanged lines ⋯ "type": "bootstrap", "title": "Adom Hydrogen (platform base)", "brief": "Layer Hydrogen's workspace onto a standard Adom workstation.",- "version": "0.5.83",+ "version": "0.5.84", "description": "Hydrogen's PLATFORM-GENERIC workspace layer on top of adom/core. Installs adom/core for the full Adom ecosystem, then adds everything common to running inside Hydrogen on ANY platform: the runtime-neutral hydrogen-* skills and the generic editor configuration (Claude Code CLI + extension, VS Code settings, trusted domains). Platform-specific layers depend on THIS: adom/hydrogen-windows-bootstrap (WSL2), adom/hydrogen-mac-bootstrap, adom/hydrogen-ubuntu-bootstrap. The reference design other orgs copy to build their own bootstrap on adom/core.", "org": "adom", "tags": [⋯ 37 unchanged lines ⋯ "auth oracle", "install grok", "grok bot",- "connect grok"+ "connect grok",+ "hydrogen runtime recovery",+ "hydrogen stuck no progress",+ "hydrogen missing app",+ "hydrogen agent loop",+ "hydrogen service broken",+ "hydrogen resume same session",+ "hydrogen stop hook diagnosis" ],- "changelog": "0.5.83: a Hydrogen workspace now has the molecule publishing skills (adom/molecule-pipeline and adom/molecule-publish), so an AI that finishes a molecule publishes it as a molecule page and the registry builds its model, footprint and symbol."+ "changelog": "0.5.84 proposed: opt-in hydrogen-runtime-recovery skill with evidence-based stop/loop diagnosis, bounded same-session recovery, verified ownership transfer, read-only defaults and offline regression tests. 0.5.83: a Hydrogen workspace now has the molecule publishing skills (adom/molecule-pipeline and adom/molecule-publish), so an AI that finishes a molecule publishes it as a molecule page and the registry builds its model, footprint and symbol." }+@@ -1,70 +1,78 @@⋯ 3 unchanged lines ⋯ "type": "bootstrap", "title": "Adom Hydrogen (platform base)", "brief": "Layer Hydrogen's workspace onto a standard Adom workstation.",- "version": "0.5.83",+ "version": "0.5.84", "description": "Hydrogen's PLATFORM-GENERIC workspace layer on top of adom/core. Installs adom/core for the full Adom ecosystem, then adds everything common to running inside Hydrogen on ANY platform: the runtime-neutral hydrogen-* skills and the generic editor configuration (Claude Code CLI + extension, VS Code settings, trusted domains). Platform-specific layers depend on THIS: adom/hydrogen-windows-bootstrap (WSL2), adom/hydrogen-mac-bootstrap, adom/hydrogen-ubuntu-bootstrap. The reference design other orgs copy to build their own bootstrap on adom/core.", "org": "adom", "tags": [⋯ 53 unchanged lines ⋯ "auth oracle", "install grok", "grok bot",- "connect grok"+ "connect grok",+ "hydrogen runtime recovery",+ "hydrogen stuck no progress",+ "hydrogen missing app",+ "hydrogen agent loop",+ "hydrogen service broken",+ "hydrogen resume same session",+ "hydrogen stop hook diagnosis" ], "changelog": "0.5.83: a Hydrogen workspace now has the molecule publishing skills (adom/molecule-pipeline and adom/molecule-publish), so an AI that finishes a molecule publishes it as a molecule page and the registry builds its model, footprint and symbol." }+@@ -1,770 +1,786 @@⋯ 45 unchanged lines ⋯ name="$(basename "$d")" install -d -m 0755 "$HOME/.claude/skills/${name}" install -m 0644 "${d}SKILL.md" "$HOME/.claude/skills/${name}/SKILL.md"+ if [ "$name" = "hydrogen-runtime-recovery" ]; then+ # Copy the complete opt-in recovery engine and its offline tests.+ # Existing bundled skills keep their original SKILL.md-only install.+ for part in tools tests; do+ if [ -d "${d}${part}" ]; then+ install -d -m 0755 "$HOME/.claude/skills/${name}/${part}"+ for src in "${d}${part}"/*.py; do+ [ -f "$src" ] || continue+ install -m 0644 "$src" "$HOME/.claude/skills/${name}/${part}/$(basename "$src")"+ done+ fi+ done+ fi count=$((count + 1)) done log " deployed ${count} bundled skills"⋯ 3 unchanged lines ⋯ rm -rf "$d" purged=$((purged + 1)) done- [ "$purged" -gt 0 ] && log " purged ${purged} stale hd-* skill dirs (renamed to hydrogen-*)"+ if [ "$purged" -gt 0 ]; then+ log " purged ${purged} stale hd-* skill dirs (renamed to hydrogen-*)"+ fi fi # 0b. Purge KNOWN-DEAD sibling module dirs. These are baked v22 leftovers whose⋯ 706 unchanged lines ⋯ HERE_W="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" mkdir -p "$HOME/.local/bin" for w in "$HERE_W"/bin/hydrogen-widget-*; do [ -f "$w" ] && install -m 0755 "$w" "$HOME/.local/bin/$(basename "$w")"; done+@@ -0,0 +1,139 @@+---+name: hydrogen-runtime-recovery+description: >-+ Detect and recover stalled Hydrogen/AI Flow tasks caused by missing apps,+ inactive services, broken Bridge connections, failed tool calls, expired+ authentication, invalid workflows or genuine software defects. Use on+ stop, no progress, repeated errors, missing tool, runtime outage, failed+ health checks, or misleading "working" status.+---++Parent skills: **hydrogen-doctor**, **hydrogen-adom-bridge-triage**, **adom-aiflow**.+Escalation skill: **wiki-contribution-assurance**.++# Hydrogen Runtime Recovery++## Purpose++A DIFFERENT skill from Contribution Assurance. This one maintains an active+engineering workflow. It discovers why Hydrogen has stopped making meaningful+progress, applies proven safe environmental recovery, and verifies the original+work item can continue from its authoritative native saved revision.++**Do not make every stop a bug report.** There are normal waits, owner holds,+long-running routing processes, expected authentication consent, completed+work and recoverable configuration mistakes.++The local deterministic sensor is `tools/triage.py`; it executes no+packet-provided commands and cannot publish, submit, merge, restart a machine,+change a PCB or spawn a competing engineering agent. It emits private incident+evidence for authorised Hydrogen orchestration.++## Trigger events++- Claude Stop hook (read-only; does not block the existing native Stop guard)+- AI Flow stage reports unfinished but bound agent becomes idle+- No physical/native board or accepted-gate progress after configured interval+- Repeated distinct failed tool invocations with the SAME blocker fingerprint+- Missing mandatory executable, app or capability+- Hydrogen Doctor shows required service down, misconfigured connection or+ broken Bridge; never mistake optional tool absence for required dependency+- Authentication expires, permissions change, or an external approval is needed+- Bad stage pointer or stale evidence appears (refer to existing AI Flow+ dependency checks rather than patching the native run JSON)+- Long-lived process exceeds normal duration: **observe and inspect** rather+ than automatically declaring it stuck.++## Triage ladder++1. Identify the authoritative AI Flow run, actual stage, accepted revision,+ current board hash and bound Claude session. Do not infer ownership from+ a similarly named project or stale dashboard.+2. Check real progress vs model activity. Agent text and tool invocations+ alone do NOT prove engineering progress. Native board revisions, gate+ decisions and file hashes do.+3. Look for an active native operation before declaring idleness. Record CPU,+ elapsed time and the last measurable milestone; do not kill legitimate+ autorouters for being slow.+4. Use existing Hydrogen Doctor (`GET /doctor`), Bridge triage,+ crash.log and wake-repair guidance. Read-only checks first.+5. Classify the blocker precisely:+ `missing_required_app`, `misconfigured_dependency`,+ `service_unhealthy`, `auth_required`, `tool_error`,+ `loop`, `idle_unfinished`, `no_progress`,+ `long_operation`, `owner_decision`, `suspected_code_defect`.+6. Repair a missing service/config only by an approved documented workflow,+ under a single-writer lease and bounded retry policy. Do not install+ random versions, bypass sign-ins, weaken electrical gates, restart the+ entire workspace or kill another agent.+7. Re-run the exact failing functional check after environmental repair.+ Compare installed versions with the EXECUTING binary path. A current+ package registry does not prove that the live pinned executable is current.+8. If there is real evidence of a software defect, create a **private**+ candidate handoff to `wiki-contribution-assurance`. Require an isolated+ reproduction, source ownership, full revision history, generic fix and+ tests. Do NOT submit anything to ADOM without explicit user approval.+9. Reconcile the original native board, stage, tool status and gates.+ Resume the SAME bound engineering work item only after proving exclusive+ ownership; avoid duplicate Claude workers and duplicate PCB operations.+10. Verify real forward progress, measure recovery outcome and record+ results privately. A plan or an idle `state:working` label is not enough.++## Decision boundaries++- `missing_required_app`: inspect platform/architecture, install source,+ version, path and authentication; prefer existing ADOM app installation+ mechanisms. Genuine optional tools may be absent without blocking work.+- `auth_required`: owner signs in; never obtain or copy secrets.+- `service_unhealthy`: Hydrogen Doctor and non-destructive wake repair+ first; workspace restart always needs a separate explicit confirmation.+- `idle_unfinished`: require unchanged native revision AND an idle bound+ agent. A new agent must not be started while another holds the board.+- `loop`: repeated DISTINCT failed operations, not repeated sampling of a+ single tool error. Fingerprint the blocker and obey cooldown.+- `suspected_code_defect`: isolated reproduction and correct owning repo+ before escalating to Contribution Assurance.+- `owner_decision`: preserve a real hold once; don't spam the owner.+- `long_operation`: inspect live process and progress; do not interrupt+ routing, copper pours or media encoding merely because their wall time+ is long.+- `completed`: do not re-open finished work.++## Commissioned private local monitor++`tools/triage.py` supplies:++```bash+python3 tools/triage.py classify --input event.json --now 1000+python3 tools/triage.py watch --config ~/.config/hydrogen-runtime-recovery/config.json \+ --state-root ~/.local/state/hydrogen-runtime-recovery --once+python3 tools/triage.py hook --config ~/.config/hydrogen-runtime-recovery/config.json \+ --state-root ~/.local/state/hydrogen-runtime-recovery+```++A local systemd user timer may call `watch` at a safe cadence (120 s).+The Stop observer only records an event; it NEVER returns a block/continue+decision, because the existing native AI Flow Stop hook owns that authority.++**Shadow mode** means automatic sensing + private incident records, NOT+automatic remediation. Do not claim fully autonomous repair or dispatch+until those controls are separately tested and explicitly enabled.++Monitor output remains under the user's private `~/.local/state` directory;+no telemetry or CAD files leave the workstation.++## Evidence and prevention++Record run ID, classified incident, native stage, progress fingerprint,+observed environmental health, relevant tool availability, exact recovery+recipe, outcome, attempt count, cooldown and whether the same original+operation succeeded.++When a generic defect is confirmed, the Contribution Assurance packet+must be built and independently checked before asking the user for+publication approval.++**Do not replace existing Hydrogen Doctor, wake-repair, Bridge triage,+AI Flow finish-guard or Hydrogen's built-in workspace monitor.**+This is an orchestrating and provenance layer that reuses them.+@@ -0,0 +1,240 @@+#!/usr/bin/env python3+"""Bounded local repair dispatcher for Hydrogen Runtime Recovery.++All mutable operations are opt-in, single-run, single-writer and idempotent.+Never submits or publishes source, never edits native PCB files, and never+relaxes AI Flow gates. The default is read-only dry-run.+"""+from __future__ import annotations+import argparse+from datetime import datetime,timezone+import fcntl+import hashlib+import json+import os+from pathlib import Path+import subprocess+import sys+import time+from urllib.parse import urlsplit+from urllib.request import Request,urlopen++def read(path):+ try:return json.loads(Path(path).read_text())+ except (OSError,ValueError):return {}++def write(path,data):+ path=Path(path);path.parent.mkdir(parents=True,exist_ok=True,mode=0o700)+ temp=path.with_name(path.name+".tmp-"+str(os.getpid()))+ with temp.open("x") as f:+ json.dump(data,f,indent=2,sort_keys=True)+ f.flush();os.fsync(f.fileno())+ os.chmod(temp,0o600);os.replace(temp,path)++def agents(command="claude"):+ try:+ p=subprocess.run([command,"agents","--json"],timeout=10,+ capture_output=True,text=True,check=False)+ if p.returncode==0:+ x=json.loads(p.stdout)+ return x if isinstance(x,list) else []+ except (OSError,ValueError,subprocess.TimeoutExpired):+ pass+ return []++def eligibility(incident,observation,config,history=None,now=None):+ """Pure decision. A classification is not authorization to mutate."""+ old=history if isinstance(history,dict) else {}+ now=time.time() if now is None else float(now)+ if not isinstance(incident,dict):return False,"no_incident"+ if config.get("shadow_mode",True):return False,"shadow_mode"+ if config.get("allow_external_submission",False):return False,"invalid_publication_policy"+ cat=incident.get("category")+ if cat=="idle_unfinished":+ if config.get("auto_dispatch") is not True:return False,"auto_dispatch_disabled"+ if incident.get("stalled_seconds",0)<config.get("min_idle_seconds",360):+ return False,"idle_below_threshold"+ if observation.get("agent_status")!="idle":return False,"agent_not_idle"+ if observation.get("active_operation") is True:return False,"native_operation_active"+ if observation.get("owner_hold") or observation.get("maintenance"):+ return False,"held"+ if observation.get("unfinished") is not True:return False,"work_not_unfinished"+ if observation.get("progress_key")!=old.get("candidate_key",observation.get("progress_key")):+ return False,"progress_changed"+ if old.get("dispatched_for_key")==observation.get("progress_key"):+ return False,"already_dispatched_for_unchanged_revision"+ attempts=[float(t) for t in old.get("resume_timestamps",[]) if+ isinstance(t,(float,int)) and 0<=now-float(t)<3600]+ budget=int(config.get("max_auto_resumes_per_hour",2))+ if budget<1 or budget>4:return False,"unsafe_resume_budget_config"+ if len(attempts)>=budget:return False,"resume_hourly_budget_exhausted"+ return True,"resume_existing_session_once"+ if cat in ("service_unhealthy","misconfigured_dependency"):+ if config.get("auto_safe_doctor_fix") is not True:return False,"doctor_fix_disabled"+ if observation.get("active_operation") is True:return False,"native_operation_active"+ if old.get("fixed_for_key")==observation.get("progress_key"):+ return False,"doctor_fix_already_attempted"+ return True,"doctor_safe_fix_once"+ return False,"not_eligible_or_owner_action"++def doctor_safe_fix(control_url):+ parsed=urlsplit(control_url)+ if parsed.scheme!="http" or parsed.hostname not in ("127.0.0.1","localhost","::1"):+ raise ValueError("Doctor URL must be verified loopback HTTP")+ uri=control_url.rstrip("/")+"/doctor"+ with urlopen(Request(uri,method="GET"),timeout=10) as f:+ before=json.loads(f.read(300000))+ b=(before.get("before") or {}) if isinstance(before,dict) else {}+ failed=int(b.get("failed") or 0)+ if failed<1:return {"status":"no_failures","before_failures":0}+ body=json.dumps({"fix":True}).encode()+ req=Request(uri,body,headers={"Content-Type":"application/json"},method="POST")+ with urlopen(req,timeout=30) as f:+ outcome=json.loads(f.read(300000))+ with urlopen(Request(uri,method="GET"),timeout=10) as f:+ after=json.loads(f.read(300000))+ checks=(after.get("before") or {}) if isinstance(after,dict) else {}+ return {+ "status":"verified" if checks.get("failed")==0 else "unresolved",+ "before_failures":failed,"after_failures":checks.get("failed"),+ "recommend_workspace_restart":bool(outcome.get("recommend_workspace_restart")),+ "actions_count":len(outcome.get("actions") or []) if isinstance(outcome.get("actions"),list) else 0,+ }++def native_guard(run_config,runner=subprocess.run):+ binary=str(run_config.get("native_aiflow_binary") or "")+ run_dir=str(run_config.get("run_dir") or "")+ if not binary or not Path(binary).is_file() or not run_dir:+ return False,"native_guard_binary_or_run_missing"+ try:+ x=runner([binary,"--run",run_dir,"finish-guard","--json"],+ timeout=18,capture_output=True,text=True)+ if x.returncode!=0:return False,"native_guard_failed"+ j=json.loads(x.stdout)+ if j.get("decision")!="continue" or j.get("mayStop") is not False:+ return False,"native_guard_does_not_authorize_continuation"+ if j.get("stopEvidence"):+ return False,"native_owner_stop_pending"+ return True,j.get("currentStep")+ except (OSError,ValueError,subprocess.TimeoutExpired,KeyError):+ return False,"native_guard_unavailable"++def run_once(config_file,state_root,apply=False,runner=subprocess.run,+ collector=None,agent_lister=None,guard=None):+ c=read(config_file)+ if c.get("schemaVersion")!="1.0":raise ValueError("bad config schema")+ events=[]+ import importlib.util+ triage_file=Path(__file__).with_name("triage.py")+ spec=importlib.util.spec_from_file_location("hr_triage",triage_file)+ triage=importlib.util.module_from_spec(spec);spec.loader.exec_module(triage)+ for run in c.get("runs",[]):+ rid=run.get("run_id")+ node=Path(state_root)/str(rid)+ node.mkdir(parents=True,exist_ok=True,mode=0o700)+ with (node/".dispatch.lock").open("a+") as f:+ fcntl.flock(f,fcntl.LOCK_EX|fcntl.LOCK_NB)+ incident=read(node/"latest.json")+ history=read(node/"dispatch-ledger.json")+ observation=(collector or triage.collect)(run)+ if (not incident or incident.get("run_id")!=rid+ or incident.get("stage")!=observation.get("stage")):+ events.append({"run_id":rid,"status":"skip","reason":"stale_incident"});continue+ if incident.get("run_state") not in ("assisted","shadow"):+ events.append({"run_id":rid,"status":"skip","reason":"unknown_incident_mode"});continue+ if incident.get("new_progress") is True:+ events.append({"run_id":rid,"status":"skip","reason":"new_progress"});continue+ if incident.get("category")=="idle_unfinished" and (observation.get("agent_status")!="idle"):+ events.append({"run_id":rid,"status":"skip","reason":"agent_resumed_elsewhere"});continue+ allowed,reason=eligibility(incident,observation,c,history)+ if not allowed or not apply:+ events.append({"run_id":rid,"status":"would_"+reason if allowed else "skip",+ "reason":reason,"applied":False});continue+ # Record dispatch intent BEFORE a potentially asynchronous subprocess.+ history["candidate_key"]=observation["progress_key"]+ history["timestamp"]=time.time()+ history["category"]=incident["category"]+ history["status"]="attempt_started"+ if reason=="resume_existing_session_once":+ history["dispatched_for_key"]=observation["progress_key"]+ attempts=[float(t) for t in history.get("resume_timestamps",[]) if+ isinstance(t,(float,int)) and 0<=time.time()-float(t)<3600]+ attempts.append(time.time())+ history["resume_timestamps"]=attempts+ else:+ history["fixed_for_key"]=observation["progress_key"]+ write(node/"dispatch-ledger.json",history)+ if reason=="resume_existing_session_once":+ sid=run.get("session_id")+ guard_ok,guard_stage=(guard or native_guard)(run,runner=runner)+ if not guard_ok or guard_stage!=observation.get("stage"):+ history["status"]="refused_native_guard"+ history["guard_reason"]=str(guard_stage)[:110]+ write(node/"dispatch-ledger.json",history)+ events.append({"run_id":rid,"status":"refused_native_guard"});continue+ observed=(agent_lister or agents)()+ owned=[a for a in observed if a.get("sessionId")==sid]+ if len(owned)!=1 or owned[0].get("status")!="idle":+ history["status"]="refused_agent_not_exclusively_idle"+ write(node/"dispatch-ledger.json",history)+ events.append({"run_id":rid,"status":"refused_agent_not_exclusively_idle"});continue+ if any(a.get("name")==owned[0].get("name") and a.get("sessionId")!=sid and a.get("status")=="working"+ for a in observed):+ history["status"]="refused_competing_agent"+ write(node/"dispatch-ledger.json",history)+ events.append({"run_id":rid,"status":"refused_competing_agent"});continue+ prompt=("Resume the authorised unfinished native ADOM AI Flow run at "++ str(run["run_dir"])+". Verify current board authority, stage and "++ "actual progress first. Diagnose missing app, auth or tool failures "++ "with Hydrogen Doctor. Investigate repeated failures using "++ "hydrogen-runtime-recovery, escalating only reproduced code defects "++ "to wiki-contribution-assurance. Work from the saved revision. "++ "Never weaken safety/manufacturing gates, create a competing board, "++ "publish to ADOM, or ask for routine engineering decisions. "++ "Commit one verified physical change or report a precise "++ "external blocker with native evidence, then proceed autonomously.")+ cmd=["claude","--bg","--resume",sid,"--permission-mode","auto",+ "--name",str(run.get("agent_name") or "Hydrogen-Existing-Run"),prompt]+ try:+ proc=runner(cmd,cwd=run.get("working_directory") or str(Path.home()/"project"),+ timeout=45,capture_output=True,text=True)+ status="resume_invoked" if proc.returncode==0 else "resume_command_failed"+ history["result"]=status+ history["exitCode"]=proc.returncode+ history["result_digest"]=hashlib.sha256((proc.stdout+proc.stderr).encode()).hexdigest()+ except (OSError,subprocess.TimeoutExpired) as e:+ status="resume_timeout_or_unavailable"+ history["result"]=status+ else:+ try:+ from urllib.parse import urlsplit+ url=str(run.get("doctor_url") or (Path.home()/".adom/hd-control-url").read_text().strip())+ info=doctor_safe_fix(url)+ history["result"]=info+ status="doctor_"+info["status"]+ except (OSError,ValueError,KeyError) as e:+ status="doctor_fix_failed"+ history["result"]=status+ history["status"]=status+ write(node/"dispatch-ledger.json",history)+ events.append({"run_id":rid,"status":status,"applied":True,+ "external_submission":False,"board_modified_by_dispatcher":False})+ return events++def main():+ ap=argparse.ArgumentParser(description="Bounded recovery; never publishes and defaults to dry-run")+ ap.add_argument("--config",type=Path,required=True)+ ap.add_argument("--state-root",type=Path,required=True)+ ap.add_argument("--apply",action="store_true",help="perform only actions explicitly enabled in local config")+ args=ap.parse_args()+ try:+ result=run_once(args.config,args.state_root,args.apply)+ print(json.dumps(result,indent=2))+ except (OSError,ValueError,RuntimeError,KeyError) as e:+ print("DISPATCH_REFUSED",str(e)[:350],file=sys.stderr)+ return 2+ return 0++if __name__=="__main__":+ raise SystemExit(main())+@@ -0,0 +1,299 @@+#!/usr/bin/env python3+"""Local Hydrogen recovery executor: deterministic gate and bounded same-session resume.++Never modifies KiCad, AI Flow, installed ADOM binaries, credentials or public services.+No PRs, Git pushes or Wiki submissions. Separate from the shadow observer.+"""+from __future__ import annotations+import argparse+import fcntl+import hashlib+import json+import os+from pathlib import Path+import subprocess+import sys+import time++CFG=Path.home()/".config/hydrogen-runtime-recovery/config.json"+STATE=Path.home()/".local/state/hydrogen-runtime-recovery"+DEFAULT_PROMPT=(+ "Continue the original authorised engineering work item using its existing "+ "Claude background session and authoritative native workflow. Confirm the "+ "exact saved project/run, current native stage, board revision, installed "+ "tools and previously accepted evidence. Read the existing project-specific "+ "engineering brief and ledger before taking action. Implement the next "+ "genuinely needed engineering change rather than ending with a plan-only "+ "handoff. Preserve all frozen technical, electrical, safety, mechanical, "+ "manufacturing and privacy constraints without waivers. Use native "+ "dry-runs, mutation postconditions and independent gate evidence; advance "+ "only upon a proven pass. Do not create a new work item, submit anything "+ "publicly or request owner approval for routine delegated engineering."+)++def jread(path):+ return json.loads(Path(path).read_text(encoding="utf8"))++def sha_file(path):+ h=hashlib.sha256()+ with open(path,"rb") as f:+ for b in iter(lambda:f.read(512*1024),b""):h.update(b)+ return h.hexdigest()++def call(*command,timeout=25,cwd=None):+ return subprocess.run(command,capture_output=True,text=True,timeout=timeout,cwd=cwd,check=False)++def collect_agents(include_stopped=False):+ cmd=("claude","agents","--all","--json") if include_stopped else ("claude","agents","--json")+ out=call(*cmd,timeout=15)+ if out.returncode:raise RuntimeError("claude agent inventory unavailable")+ j=json.loads(out.stdout)+ if not isinstance(j,list):raise RuntimeError("invalid claude agent inventory")+ return [x for x in j if isinstance(x,dict)]++def validate(config):+ if config.get("schemaVersion")!="1.0" or config.get("allow_external_submission") is not False:+ raise RuntimeError("unknown or externally enabled configuration")+ if config.get("auto_dispatch") is not False:+ raise RuntimeError("other dispatch policy already enabled; refuse competing controls")+ runs=config.get("runs")+ if not isinstance(runs,list):raise RuntimeError("run bindings missing")+ return runs++def quote_ident(v):+ return isinstance(v,str) and 3<=len(v)<=80 and all(c.isalnum() or c in "_-" for c in v)++def attest(config,run_id,now=None,check_doctor=True):+ now=time.time() if now is None else now+ runs=validate(config)+ targets=[r for r in runs if r.get("run_id")==run_id]+ if len(targets)!=1 or not quote_ident(run_id):+ return {"approved":False,"reasons":["run not bound exactly once"],"run_id":run_id}+ r=targets[0]+ directory=Path(str(r.get("run_dir","")))+ board=Path(str(r.get("board_path","")))+ sid=r.get("session_id")+ if not directory.is_absolute() or not board.is_absolute() or directory.name!="run" or not sid:+ return {"approved":False,"reasons":["invalid native run, board, or session binding"],"run_id":run_id}+ reasons=[]+ runfile=directory/"run.json"+ if not runfile.is_file() or not board.is_file():+ reasons.append("missing authoritative native board/run")+ return {"approved":False,"reasons":reasons,"run_id":run_id}+ native=jread(runfile)+ stage=native.get("currentStep")+ if not isinstance(stage,str) or not stage or stage not in native.get("sequence",[]):+ reasons.append("invalid AI Flow current stage or run sequence")+ if native.get("status") in ("completed","delivered") or native.get("complete") is True:+ reasons.append("native AI Flow is complete")+ journal=STATE/run_id/"latest.json"+ if not journal.is_file():+ reasons.append("no observer evidence")+ observed={}+ else:+ observed=jread(journal)+ category=observed.get("category")+ if category not in ("idle_unfinished","no_progress"):+ reasons.append("observer does not confirm idle or stale unfinished work")+ if observed.get("run_state")!="shadow" or observed.get("stage")!=stage:+ reasons.append("observer run/stage disagreement")+ if observed.get("may_publish") is not False or observed.get("may_spawn_agent") is not False:+ reasons.append("observer privileges unexpectedly enabled")+ if observed.get("timestamp"):+ try:+ from datetime import datetime+ age=now-datetime.fromisoformat(observed["timestamp"]).timestamp()+ if not -30<=age<=600:reasons.append("observer evidence stale")+ except ValueError:reasons.append("malformed observer timestamp")+ else:reasons.append("observer timestamp absent")+ agents=collect_agents()+ historical=collect_agents(include_stopped=True)+ selected=[a for a in historical if a.get("sessionId")==sid]+ if len(selected)!=1:+ reasons.append("bound agent missing or duplicated")+ else:+ a=selected[0]+ safe_stopped=a.get("state")=="stopped" and a.get("status") in (None,"stopped")+ safe_idle=a.get("status")=="idle" and a.get("state") in ("working","idle",None)+ if a.get("kind")!="background" or not (safe_idle or safe_stopped):+ reasons.append("bound agent is not idle or stopped background session")+ if category=="no_progress" and not safe_stopped:+ reasons.append("no_progress recovery requires the bound background session already stopped")+ if not str(a.get("id") or "").isalnum():+ reasons.append("background CLI job id missing/unsafe")+ preferred_cwd=r.get("session_cwd")+ if preferred_cwd:+ cwd_path=Path(str(preferred_cwd))+ if not cwd_path.is_absolute() or not cwd_path.is_dir() or str(cwd_path)!=a.get("cwd"):+ reasons.append("configured session working directory differs from the original agent")+ active=[a.get("sessionId") for a in agents if a.get("sessionId")!=sid and a.get("status") in ("working","running","busy","active")]+ if active:reasons.append("another agent is active; cannot guarantee exclusive board ownership")+ from importlib import util+ triage_path=Path.home()/".claude/skills/hydrogen-runtime-recovery/tools/triage.py"+ if not triage_path.is_file():+ reasons.append("trusted runtime monitor missing")+ act=[]+ else:+ spec=util.spec_from_file_location("triage",triage_path)+ tr=util.module_from_spec(spec);spec.loader.exec_module(tr)+ collected=tr.collect(r)+ act=collected.get("active_process_markers",[])+ if collected.get("active_operation"):+ reasons.append("native engineering operation currently active")+ if collected.get("progress_key") is None:+ reasons.append("native progress fingerprint missing")+ if check_doctor:+ checks=tr.capture_doctor(r)+ if len(checks)<10:reasons.append("Hydrogen Doctor returned insufficient checks")+ failed=[c.get("id") for c in checks if c.get("ok") is False]+ if failed:reasons.append("Hydrogen Doctor checks failed: "+",".join(map(str,failed[:7])))+ if now-runfile.stat().st_mtime<240:+ reasons.append("native run updated too recently to treat as stuck")+ if now-board.stat().st_mtime<240:+ reasons.append("board updated too recently to treat as stuck")+ brief=r.get("work_item_brief")+ if brief:+ path=Path(str(brief))+ project_root=directory.parent.parent.resolve()+ if (not path.is_absolute() or not path.is_file() or path.is_symlink()+ or not path.resolve().is_relative_to(project_root)):+ reasons.append("configured work item brief missing or outside owning project")+ history_path=STATE/run_id/"recovery-attempts.json"+ history=jread(history_path) if history_path.is_file() else {"attempts":[]}+ if not isinstance(history.get("attempts"),list):raise RuntimeError("invalid local attempt ledger")+ revision_fingerprint=hashlib.sha256((sha_file(runfile)+sha_file(board)).encode()).hexdigest()+ matching=[a for a in history["attempts"]+ if a.get("fingerprint")==revision_fingerprint+ and a.get("status") in ("launch_requested","launch_failed","unverified_session")]+ if len(matching)>=2:+ reasons.append("retry budget exhausted for unchanged native revision")+ if matching and now-float(matching[-1].get("time",0))<900:+ reasons.append("recovery cooldown has not elapsed")+ return {+ "approved":not reasons,+ "reasons":reasons,+ "run_id":run_id,"stage":stage,"session_id":sid,+ "work_item_brief":str(brief or ""),+ "session_cwd":str(r.get("session_cwd") or+ (selected[0].get("cwd") if len(selected)==1 else "") or directory.parent.parent),+ "background_id":selected[0].get("id") if len(selected)==1 else None,+ "bound_session_stopped":bool(selected and selected[0].get("state")=="stopped"),+ "revision_fingerprint":revision_fingerprint,+ "native_run_sha256":sha_file(runfile),"native_board_sha256":sha_file(board),+ "checked_at":now,+ "agent_statuses":[{"sessionId":a.get("sessionId"),"kind":a.get("kind"),+ "status":a.get("status")} for a in agents],+ "active_process_markers":act,"previous_attempts_at_revision":len(matching),+ "permitted_next_action":("resume_same_session" if not reasons else "investigate"),+ "may_publish":False,"may_relax_gates":False}++def atomic(path,obj):+ p=Path(path);p.parent.mkdir(parents=True,exist_ok=True,mode=0o700)+ tmp=p.with_name(p.name+".tmp"+str(os.getpid()))+ try:+ fd=os.open(tmp,os.O_CREAT|os.O_WRONLY|os.O_EXCL,0o600)+ with os.fdopen(fd,"w") as f:+ json.dump(obj,f,indent=2);f.flush();os.fsync(f.fileno())+ os.replace(tmp,p)+ finally:+ if tmp.exists():tmp.unlink()++def execute(config,run_id,planfile=None,prompt=DEFAULT_PROMPT):+ node=STATE/run_id;node.mkdir(parents=True,exist_ok=True,mode=0o700)+ with (node/"exclusive-resume.lock").open("a+") as f:+ fcntl.flock(f,fcntl.LOCK_EX|fcntl.LOCK_NB)+ pre=attest(config,run_id)+ if not pre["approved"]:+ return {"executed":False,"reason":"preflight_blocked","reasons":pre["reasons"]}+ if planfile is not None:+ presented=jread(planfile)+ for field in ("revision_fingerprint","session_id","stage", "native_board_sha256","native_run_sha256","background_id","work_item_brief","session_cwd"):+ if presented.get(field)!=pre.get(field):+ return {"executed":False,"reason":"stale_approved_plan","field":field}+ ledger=node/"recovery-attempts.json"+ history=jread(ledger) if ledger.exists() else {"attempts":[]}+ attempt={"time":time.time(),"stage":pre["stage"],"fingerprint":pre["revision_fingerprint"],+ "session_id":pre["session_id"],"operation":"resume_existing_idle_session",+ "status":"reserved","owner":"one native run","public_submission":False}+ history["attempts"].append(attempt)+ atomic(ledger,history)+ sid=pre["session_id"]+ # Claude's own documented CLI reuses the same ID only when it is+ # NOT already running. Explicitly retire the confirmed idle holder first.+ if not pre.get("bound_session_stopped"):+ jobid=pre["background_id"]+ old=call("claude","stop",jobid,timeout=20)+ if old.returncode:+ attempt.update(status="stop_failed",diagnostic=(old.stderr or old.stdout)[-350:])+ atomic(ledger,history)+ return {"executed":False,"reason":"claude_stop_failed"}+ agents=collect_agents()+ if any(a.get("sessionId")==sid and a.get("status") in ("working","running","busy","active") for a in agents):+ attempt.update(status="old_session_still_active")+ atomic(ledger,history)+ return {"executed":False,"reason":"still_active_after_stop"}+ # Fence after agent-stop, before any new engineering instruction.+ native_path=next(x["run_dir"] for x in config["runs"] if x.get("run_id")==run_id)+ board_path=next(x["board_path"] for x in config["runs"] if x.get("run_id")==run_id)+ if sha_file(Path(native_path)/"run.json")!=pre["native_run_sha256"] or sha_file(board_path)!=pre["native_board_sha256"]:+ attempt.update(status="source_changed_before_resume")+ atomic(ledger,history)+ return {"executed":False,"reason":"native_revision_changed"}+ # This invokes the *existing* Claude session without --fork-session.+ # Never interprets a CLI request success as a completed PCB action.+ project_root=str(Path(native_path).resolve().parent.parent)+ brief=pre.get("work_item_brief") or ""+ brief_instruction=(" Read the owner-approved private engineering brief at: "++ brief+"." if brief else "")+ full_prompt=(prompt+" The authoritative native run is "+native_path++ "; current stage is "+pre["stage"]+"."+brief_instruction)+ session_cwd=pre.get("session_cwd")+ if not session_cwd or not Path(session_cwd).is_dir():+ attempt.update(status="session_cwd_invalid")+ atomic(ledger,history)+ return {"executed":False,"reason":"session_cwd_invalid"}+ started=call("claude","--bg","--resume",sid,"--permission-mode","auto",+ "--name","Recovery-"+run_id,full_prompt,+ timeout=65,cwd=session_cwd)+ attempt["launcher_exit"]=started.returncode+ attempt["launcher_response"]=(started.stdout or started.stderr)[-450:]+ attempt["status"]="launch_requested" if started.returncode==0 else "launch_failed"+ atomic(ledger,history)+ agents=collect_agents()+ same=[a for a in agents if a.get("sessionId")==sid]+ attempt["same_session_visible"]=bool(same)+ attempt["same_session_status"]=[a.get("status") for a in same]+ if started.returncode==0 and not same:+ attempt["status"]="unverified_session"+ atomic(ledger,history)+ return {"executed":started.returncode==0 and bool(same),+ "reason":attempt["status"],"same_session_visible":bool(same),+ "current_native_stage":pre["stage"],+ "engineering_gate_passed":False,+ "verify_board_progress":True,"may_publish":False}++def main():+ cli=argparse.ArgumentParser()+ cli.add_argument("action",choices=["preflight","resume"])+ cli.add_argument("--run-id",default="hsc01")+ cli.add_argument("--output")+ cli.add_argument("--plan-file")+ cli.add_argument("--no-doctor",action="store_true")+ args=cli.parse_args()+ config=jread(CFG)+ if args.action=="preflight":+ result=attest(config,args.run_id,check_doctor=not args.no_doctor)+ else:+ if not args.plan_file:+ raise SystemExit("resume requires --plan-file from a recent preflight")+ result=execute(config,args.run_id,args.plan_file)+ if args.output:atomic(args.output,result)+ print(json.dumps(result,indent=2))+ return 0 if (result.get("approved") if args.action=="preflight" else result.get("executed")) else 1++if __name__=="__main__":+ try:raise SystemExit(main())+ except Exception as e:+ print("RECOVERY_GATE_ERROR",type(e).__name__,str(e)[:350],file=sys.stderr)+ raise SystemExit(2)+@@ -0,0 +1,491 @@+#!/usr/bin/env python3+"""Hydrogen Runtime Recovery: private, bounded, evidence-based stalled-work triage.++This is NOT an autonomous publisher or AI prompt dispatcher. The default+action is a local diagnosis and a durable event for the same Hydrogen run.+"""+from __future__ import annotations+import argparse+from datetime import datetime, timezone+import hashlib+import json+import os+from pathlib import Path+import re+import shutil+import subprocess+import sys+import time++CATEGORIES={+ "healthy","working","long_operation","no_progress","idle_unfinished",+ "loop","missing_required_app","misconfigured_dependency","service_unhealthy",+ "auth_required","owner_decision","tool_error","suspected_code_defect",+ "environment_restored","paused","completed"+}+AUTH_RE=re.compile(r"(?i)(401|403|unauthori[sz]ed|session.expired|login.required|token.expired)")+MISSING_RE=re.compile(r"(?i)(command.not.found|ENOENT|no such file|executable.not.found|module.not.found)")+CONFIG_RE=re.compile(r"(?i)(invalid.config|missing.environment|missing.variable|misconfigured)")+ERROR_RE=re.compile(r"(?i)(error|failed|panic|exception)")+DEFAULT_POLICY={+ "idle_seconds":360,+ "no_progress_seconds":900,+ "long_operation_seconds":1800,+ "loop_repeats":3,+ "cooldown_seconds":1200,+ "max_history":32+}+REMEDIATION={+ "missing_required_app":("environment","hydrogen-setup-steps","Check installed path, package registry and executable before installing; no hidden installs"),+ "misconfigured_dependency":("environment","hydrogen-doctor","Verify configuration and permissions, use documented non-destructive repair"),+ "service_unhealthy":("environment","hydrogen-doctor","Read GET /doctor; only non-disruptive Doctor fixes when explicitly enabled"),+ "auth_required":("owner","hydrogen-adom-auth","Report which connection expired; never extract secrets or bypass sign-in"),+ "owner_decision":("owner","hydrogen-permissions","Stop once with precise external decision required; no repeated requests"),+ "tool_error":("investigate","hydrogen-adom-bridge-triage","Read actual error, installed version and latest crash logs; reproduce"),+ "suspected_code_defect":("assurance","wiki-contribution-assurance","Require minimal reproduction on isolated fixture before an upstream fix"),+ "loop":("investigate","hydrogen-skill-loop","Inspect repeated identical failures and unchanged revisions; do not blindly retry"),+ "no_progress":("investigate","hydrogen-doctor","Check active tool, native run and provider before considering resumption"),+ "idle_unfinished":("continue","adom-aiflow","Verify original work-item ownership and resume same session; never spawn duplicate writer"),+ "long_operation":("observe","hydrogen-monitor","Inspect active process and milestones, preserve long legitimate work"),+ "working":("observe","hydrogen-monitor","No intervention"),+ "healthy":("observe","hydrogen-monitor","No intervention"),+ "environment_restored":("continue","adom-aiflow","Recheck native gates and resume only the bound work item"),+ "completed":("none","adom-aiflow","No intervention"),+ "paused":("none","hydrogen-permissions","Explicit hold respected"),+}+SAFE_KINDS={"service_unhealthy","misconfigured_dependency"}++def digest_data(value):+ return hashlib.sha256(json.dumps(value,sort_keys=True,separators=(',',':'),default=str).encode()).hexdigest()++def observation_value(x,key,default=None):+ return x.get(key,default) if isinstance(x,dict) else default++def evaluate(observation,previous=None,policy=None,now=None):+ """Pure classification; no tool execution or mutation."""+ if not isinstance(observation,dict):+ raise ValueError("observation must be an object")+ p=dict(DEFAULT_POLICY)+ if policy:+ for k in DEFAULT_POLICY:+ if k in policy:+ v=policy[k]+ if type(v) is not int or v < 1 or v > 604800:+ raise ValueError(f"invalid policy value {k}")+ p[k]=v+ now=float(time.time() if now is None else now)+ old=previous if isinstance(previous,dict) else {}+ key=str(observation.get("progress_key") or "")+ if not key:+ raise ValueError("progress_key is required and must be nonempty")+ prior_key=old.get("progress_key")+ progressed=prior_key!=key+ first=now if progressed else min(now,float(old.get("progress_since",now)))+ stalled_for=max(0,now-first)+ event=observation.get("event") or {}+ if not isinstance(event,dict):event={}+ failure=str(event.get("code") or event.get("message") or "")+ fail_fingerprint=digest_data([failure,observation.get("run_id"),observation.get("stage")]) if failure else ""+ event_id=str(event.get("event_id") or "")+ same_failure=failure and fail_fingerprint==old.get("failure_fingerprint") and not progressed+ same_event=bool(event_id) and event_id==old.get("last_event_id")+ repeats=(int(old.get("repeats",0)) if same_event and same_failure+ else int(old.get("repeats",0))+1 if same_failure+ else 1 if failure else 0)+ incomplete=observation.get("unfinished") is True+ active=observation.get("active_operation") is True+ agent=str(observation.get("agent_status") or "unknown")+ declared=str(observation.get("declared_state") or "")+ checks=observation.get("doctor_checks") or []+ if not isinstance(checks,list):checks=[]+ failures=[c for c in checks if isinstance(c,dict) and c.get("required",True) and c.get("ok") is False]+ missing=[x for x in observation.get("required_tools",[]) if isinstance(x,dict) and x.get("required",True) and x.get("available") is False]+ kind="healthy"+ evidence=[]+ if observation.get("completed") is True:+ kind="completed"+ elif observation.get("owner_hold") is True or event.get("kind")=="owner_decision":+ kind="owner_decision"+ elif observation.get("paused") is True or observation.get("maintenance") is True:+ kind="paused"+ elif AUTH_RE.search(failure) or any(c.get("kind")=="authentication" for c in failures):+ kind="auth_required"+ elif missing:+ kind="missing_required_app";evidence=[x.get("name","unknown") for x in missing]+ elif MISSING_RE.search(failure):+ kind="missing_required_app";evidence=["command/executable mentioned in recent failure"]+ elif CONFIG_RE.search(failure) or any(c.get("kind")=="configuration" for c in failures):+ kind="misconfigured_dependency"+ elif failures:+ kind="service_unhealthy";evidence=[str(c.get("id",c.get("name","doctor"))) for c in failures]+ elif event.get("reproduced_defect") is True and event.get("isolated_fixture") is True:+ kind="suspected_code_defect"+ elif failure and repeats>=p["loop_repeats"] and not progressed:+ kind="loop"+ elif failure:+ kind="tool_error"+ elif agent in ("busy","working") and isinstance(observation.get("recent_tool_at"),(int,float)) and 0 <= now-observation["recent_tool_at"] <= 600:+ kind="working"+ elif active and stalled_for >= p["long_operation_seconds"]:+ kind="long_operation"+ elif active:+ kind="working"+ elif incomplete and agent=="idle" and (declared=="working" or stalled_for>=p["idle_seconds"]) and stalled_for>=p["idle_seconds"]:+ kind="idle_unfinished"+ elif incomplete and stalled_for>=p["no_progress_seconds"]:+ kind="no_progress"+ elif incomplete:+ kind="working" if agent=="working" else "healthy"+ else:+ kind="healthy"+ route,skill,action=REMEDIATION[kind]+ fingerprint=digest_data([observation.get("run_id"),kind,key,failure,tuple(evidence)])+ same=fingerprint==old.get("decision_fingerprint")+ cooldown=max(0,p["cooldown_seconds"]-(now-float(old.get("last_alert",0)))) if same else 0+ alert=(kind not in ("healthy","working","completed","paused") and cooldown==0)+ result={+ "schemaVersion":"1.0",+ "run_id":str(observation.get("run_id") or ""),+ "stage":str(observation.get("stage") or ""),+ "category":kind,"route":route,"skill":skill,+ "recommended_action":action,+ "progress_since":first,"stalled_seconds":round(stalled_for,2),+ "new_progress":progressed,"same_failure_repeats":repeats,+ "decision_fingerprint":fingerprint,+ "alert":alert,"suppressed_by_cooldown":bool(cooldown),+ "evidence":evidence,+ "safe_autoheal_eligible":kind in SAFE_KINDS and not active,+ "may_publish":False,"may_spawn_agent":False,"may_relax_gates":False+ }+ state={+ "progress_key":key,"progress_since":first,+ "failure_fingerprint":fail_fingerprint,"last_event_id":event_id,"repeats":repeats,+ "decision_fingerprint":fingerprint,+ "last_alert":now if alert else float(old.get("last_alert",0)),+ "last_observed":now,"last_category":kind,+ }+ return result,state++def json_file(path,default):+ try:return json.loads(Path(path).read_text(encoding="utf8"))+ except (OSError,ValueError,TypeError):return default++def atomic_json(path,payload):+ p=Path(path)+ p.parent.mkdir(parents=True,exist_ok=True,mode=0o700)+ tmp=p.with_name(p.name+".tmp-"+str(os.getpid()))+ try:+ fd=os.open(tmp,os.O_CREAT|os.O_EXCL|os.O_WRONLY,0o600)+ with os.fdopen(fd,"w",encoding="utf8") as f:+ json.dump(payload,f,ensure_ascii=False,sort_keys=True,indent=2)+ f.flush();os.fsync(f.fileno())+ os.replace(tmp,p)+ finally:+ if tmp.exists():tmp.unlink()++def latest_tool_activity(config,now=None):+ """Return recent real tool invocation time; copied historical context ignored."""+ sid=config.get("session_id")+ journal=config.get("journal_path")+ if not journal and sid:+ matches=list((Path.home()/".claude"/"projects").glob(f"*/{sid}.jsonl"))+ journal=matches[0] if len(matches)==1 else None+ if not journal:return None+ path=Path(journal)+ if path.is_symlink():return None+ try:+ with path.open("rb") as stream:+ stream.seek(0,2)+ size=stream.tell()+ stream.seek(max(0,size-700000))+ if size>700000:stream.readline()+ lines=stream.readlines()+ except OSError:return None+ now=time.time() if now is None else now+ for raw in reversed(lines[-1500:]):+ try:item=json.loads(raw)+ except ValueError:continue+ if item.get("type")!="assistant":continue+ blocks=(item.get("message") or {}).get("content",[]) or []+ if not any(isinstance(z,dict) and z.get("type")=="tool_use" for z in blocks):+ continue+ try:+ t=datetime.fromisoformat(str(item.get("timestamp","")).replace("Z","+00:00")).timestamp()+ except ValueError:continue+ if now-900<=t<=now+30:return t+ return None++def read_recent_tool_error(config,now=None):+ """Read recent FAILED tool results only; never retain tool output or secrets."""+ sid=config.get("session_id")+ if not sid:return {}+ journal=config.get("journal_path")+ if not journal:+ candidates=list((Path.home()/".claude"/"projects").glob(f"*/{sid}.jsonl"))+ journal=candidates[0] if len(candidates)==1 else None+ if not journal:return {}+ file=Path(journal)+ if file.is_symlink():return {}+ try:+ with file.open("rb") as f:+ f.seek(0,2);sz=f.tell()+ f.seek(max(0,sz-500000))+ if sz>500000:f.readline()+ lines=f.readlines()+ except OSError:+ return {}+ now=float(time.time() if now is None else now)+ for raw in reversed(lines[-1000:]):+ try:j=json.loads(raw)+ except ValueError:continue+ if j.get("type")!="user":continue+ try:+ stamp=datetime.fromisoformat(str(j.get("timestamp","")).replace("Z","+00:00")).timestamp()+ except ValueError:continue+ if stamp<now-600 or stamp>now+30:continue+ for c in (j.get("message") or {}).get("content",[]) or []:+ if not isinstance(c,dict) or c.get("type")!="tool_result" or c.get("is_error") is not True:continue+ raw_body=str(c.get("content") or "")+ if AUTH_RE.search(raw_body):kind="401 authentication required"+ elif MISSING_RE.search(raw_body):kind="command not found"+ elif CONFIG_RE.search(raw_body):kind="invalid configuration"+ else:kind="tool invocation error"+ return {"code":kind,"event_id":str(c.get("tool_use_id") or j.get("uuid") or stamp)[:125],+ "timestamp":stamp}+ return {}++def identify_active_engineering_processes(ps_text,markers):+ """Never count a timeline server, standby server or grep as CAD progress."""+ supported={"adom-aiflow","kicad-cli","java","python3"}+ infrastructure=("timeline serve","standby-server","-m http.server","--help","watch --config")+ working=()+ hits=[]+ for raw in str(ps_text).splitlines():+ fields=raw.strip().split(None,1)+ if len(fields)!=2:continue+ comm,args=fields+ if comm not in supported:continue+ if any(x in args for x in infrastructure):continue+ if comm=="adom-aiflow":+ # A live ADOM CLI in read-only timeline serve mode is NEVER a route.+ # Only finite engineering operations count as active work.+ if not re.search(r"\s(route|land|gate|pours|placement|fields|current|thermal|verify)(\s|$)",args):+ continue+ elif comm=="java" and "freerouting" not in args.lower():+ continue+ elif comm=="python3" and "kicad_route_net" not in args:+ continue+ for marker in markers:+ if isinstance(marker,str) and marker and marker in args:+ hits.append(marker)+ break+ return sorted(set(hits))++def collect(config):+ """Read-only local collector; does not execute user-provided shell strings."""+ run=Path(config["run_dir"]).expanduser().resolve()+ j=json_file(run/"run.json",{})+ if not isinstance(j,dict):j={}+ stage=str(j.get("currentStep") or "")+ board=Path(config.get("board_path") or run/"board.kicad_pcb").expanduser()+ pstat=None+ try:+ s=board.stat()+ pstat=(s.st_mtime_ns,s.st_size)+ except OSError:pass+ # Stage and board modification count as physical or state progress.+ progress_key=digest_data([stage,pstat,j.get("position"),j.get("currentRevision"),j.get("resultHash")])+ agents=[]+ try:+ result=subprocess.run(["claude","agents","--json"],capture_output=True,text=True,timeout=10,check=False)+ if result.returncode==0:agents=json.loads(result.stdout)+ except (OSError,ValueError,subprocess.TimeoutExpired):+ pass+ if not isinstance(agents,list):agents=[]+ session_id=config.get("session_id")+ selected=next((a for a in agents if isinstance(a,dict) and a.get("sessionId")==session_id),{})+ commands=[]+ try:+ result=subprocess.run(["ps","-eo","comm=,args="],capture_output=True,text=True,timeout=3)+ if result.returncode==0:+ commands=identify_active_engineering_processes(+ result.stdout,config.get("active_command_markers",[]))+ except (OSError,subprocess.TimeoutExpired):pass+ tools=[]+ for tool in config.get("required_tools",[]):+ if not isinstance(tool,dict):continue+ name=str(tool.get("name",""))+ target=tool.get("path")+ ok=bool(shutil.which(name)) if not target else (Path(target).exists() and os.access(target,os.X_OK))+ tools.append({"name":name,"available":ok,"required":tool.get("required",True)})+ incomplete=bool(stage) and j.get("status") not in ("completed","delivered") and j.get("complete") is not True+ return {"run_id":config.get("run_id",run.name),"stage":stage,+ "unfinished":incomplete,"completed":not incomplete,+ "progress_key":progress_key,+ "agent_status":selected.get("status","unknown"),"declared_state":selected.get("state",""),+ "active_operation":bool(commands),"active_process_markers":commands,+ "required_tools":tools,"doctor_checks":[],"event":read_recent_tool_error(config),+ "recent_tool_at":latest_tool_activity(config),+ "observed_at":time.time(),"session_id_found":bool(selected)}++def capture_doctor(config):+ """Read existing Hydrogen Doctor; ignore secrets and untrusted remote URLs."""+ if not config.get("doctor_enabled",False):+ return []+ from urllib.request import Request,urlopen+ from urllib.parse import urlsplit+ url=str(config.get("doctor_url") or "")+ if not url:+ try:url=(Path.home()/".adom/hd-control-url").read_text().strip()+ except OSError:url=""+ parsed=urlsplit(url)+ if parsed.scheme!="http" or parsed.hostname not in ("127.0.0.1","localhost","::1"):+ return [{"id":"doctor_url","ok":False,"kind":"configuration","required":True}]+ try:+ with urlopen(Request(url.rstrip("/")+"/doctor",method="GET"),timeout=8) as response:+ payload=json.loads(response.read(300000))+ before=payload.get("before") or {} if isinstance(payload,dict) else {}+ checks=before.get("checks",[]) if isinstance(before,dict) else []+ if isinstance(checks,dict):+ checks=[{"id":k,**(v if isinstance(v,dict) else {"ok":v is True})}+ for k,v in checks.items()]+ return [{"id":str(c.get("id") or c.get("check") or c.get("name") or "?")[:72],+ "ok":c.get("ok"),"kind":str(c.get("kind") or "service")[:30],+ "required":c.get("required",True)}+ for c in checks if isinstance(c,dict)][:70]+ except Exception:+ return [{"id":"doctor_unreachable","ok":False,"kind":"service","required":True}]++def validate_config(config):+ if not isinstance(config,dict) or config.get("schemaVersion")!="1.0":+ raise ValueError("config must use schemaVersion 1.0")+ if not isinstance(config.get("runs"),list) or not config["runs"]:+ raise ValueError("config.runs must be nonempty")+ seen=set()+ for run in config["runs"]:+ if not isinstance(run,dict):+ raise ValueError("run config must be an object")+ ident=str(run.get("run_id") or "")+ if not re.fullmatch(r"[A-Za-z0-9_-]{3,80}",ident) or ident in seen:+ raise ValueError("run_id must be unique and safe for local state paths")+ seen.add(ident)+ directory=Path(str(run.get("run_dir") or ""))+ if not directory.is_absolute() or directory.name!="run":+ raise ValueError("run_dir must be an absolute native AI Flow run directory")+ if not re.fullmatch(r"[a-f0-9-]{36}",str(run.get("session_id") or "")):+ raise ValueError("session_id must be an explicit UUID")+ return config++def action_watch(config_file,root,policy=None,now=None,ephemeral=False):+ import fcntl+ c=validate_config(json_file(config_file,{}))+ root=Path(root).expanduser()+ root.mkdir(parents=True,exist_ok=True,mode=0o700)+ report=[]+ with (root/".single-writer.lock").open("a+") as lock:+ fcntl.flock(lock,fcntl.LOCK_EX)+ for run in c["runs"]:+ rid=run["run_id"]+ node=root/rid+ node.mkdir(parents=True,exist_ok=True,mode=0o700)+ o=collect(run)+ o["doctor_checks"]=capture_doctor(run)+ base=0+ try:+ files=[Path(run["run_dir"])/"run.json"]+ if run.get("board_path"):files.append(Path(run["board_path"]))+ base=max(f.stat().st_mtime for f in files if f.is_file())+ except (OSError,ValueError):+ base=0+ if base and not o.get("initial_progress_since"):+ o["initial_progress_since"]=base+ hook=node/"last-hook.json"+ recorded=json_file(hook,{})+ if (recorded.get("session_id")==run["session_id"]+ and float(recorded.get("received_at",0))>=time.time()-600):+ o["last_stop_hook_at"]=recorded["received_at"]+ old=json_file(node/"state.json",{})+ current,updated=evaluate(o,old,policy or c.get("policy"),now=now)+ if not old.get("progress_key") and o.get("initial_progress_since"):+ # On first observation, infer an approximate last genuine change+ # from the saved native board/run mtime, not agent text activity.+ updated["progress_since"]=min(float(now or time.time()),o["initial_progress_since"])+ current,updated=evaluate(o,{**old,"progress_key":o["progress_key"],+ "progress_since":updated["progress_since"]},policy or c.get("policy"),now=now)+ current["timestamp"]=datetime.now(timezone.utc).isoformat()+ current["run_state"]="shadow" if c.get("shadow_mode",True) else "assisted"+ if not ephemeral:+ atomic_json(node/"state.json",updated)+ atomic_json(node/"latest.json",current)+ if current["alert"]:+ entry=node/"incidents"/(str(int(now or time.time()))+"-"+current["decision_fingerprint"][:12]+".json")+ atomic_json(entry,current)+ if current["category"]=="suspected_code_defect":+ # This record is only a handoff suggestion, NOT a PR or a+ # claim that a true defect has been reproduced upstream.+ atomic_json(node/"contribution-assurance-handoff.json",{+ "classification":"unverified","may_publish":False,+ "status":"investigation_required",+ "case_id":current["decision_fingerprint"],+ "cause":"reproduced defect indicated; independent source and fixture audit required",+ "next_skill":"wiki-contribution-assurance"})+ report.append(current)+ return report++def action_hook(config_file,root,stdin):+ """Nonblocking Claude Stop observer: preserve existing stop guard authority."""+ c=validate_config(json_file(config_file,{}))+ try:o=json.loads(stdin[:50000])+ except (TypeError,ValueError):return 0+ if not isinstance(o,dict) or o.get("hook_event_name") not in ("Stop",None):+ return 0+ sid=o.get("session_id")+ for run in c["runs"]:+ if run.get("session_id")!=sid:+ continue+ node=Path(root).expanduser()/run["run_id"]+ atomic_json(node/"last-hook.json",{+ "received_at":time.time(),"event":"Stop",+ "session_id":sid,"owner":"same_session_only",+ "no_block":True,"no_dispatch":True})+ break+ return 0++def main(argv=None):+ cli=argparse.ArgumentParser(description="Hydrogen Runtime Recovery; private triage, never submits or edits board files")+ subs=cli.add_subparsers(dest="action",required=True)+ inspect=subs.add_parser("classify",help="Evaluate a synthetic or captured observation")+ inspect.add_argument("--input",required=True,type=Path)+ inspect.add_argument("--previous",type=Path)+ inspect.add_argument("--now",type=float)+ inspect.add_argument("--out",type=Path)+ watcher=subs.add_parser("watch",help="Collect live read-only state, evaluate and record incidents")+ watcher.add_argument("--config",required=True,type=Path)+ watcher.add_argument("--state-root",required=True,type=Path)+ watcher.add_argument("--once",action="store_true")+ hook=subs.add_parser("hook",help="Nonblocking observation from a Claude Stop hook")+ hook.add_argument("--config",required=True,type=Path)+ hook.add_argument("--state-root",required=True,type=Path)+ args=cli.parse_args(argv)+ try:+ if args.action=="classify":+ current,state=evaluate(json_file(args.input,{}),+ json_file(args.previous,{}) if args.previous else None,now=args.now)+ if args.out:atomic_json(args.out,current)+ print(json.dumps({"classification":current,"nextState":state},indent=2))+ return 0+ if args.action=="watch":+ result=action_watch(args.config,args.state_root)+ print(json.dumps(result,indent=2))+ return 0+ if args.action=="hook":+ # Hooks must NEVER block the engineering agent or print an instruction.+ return action_hook(args.config,args.state_root,sys.stdin.read(50000))+ except (OSError,ValueError,TypeError,KeyError) as e:+ print("RECOVERY_MONITOR_ERROR:",str(e)[:350],file=sys.stderr)+ return 2+ return 2++if __name__=="__main__":+ raise SystemExit(main())+@@ -0,0 +1,176 @@+#!/usr/bin/env python3+"""Safe validation and private ownership transfer for Claude background resume forks.++This module has no launcher, CAD editor, remote upload or publishing code.+"""+from __future__ import annotations+from datetime import datetime,timezone+import fcntl+import hashlib+import json+import os+from pathlib import Path+import re+import time++SHORT_ID=re.compile(r"\bbackgrounded\s*[·:|\-]\s*([0-9a-f]{8})\b",re.I)+UUID=re.compile(r"^[a-f0-9]{8}-[a-f0-9-]{27}$",re.I)++def launch_job_id(stdout):+ m=SHORT_ID.search(str(stdout or ""))+ return m.group(1).lower() if m else None++def get_prompt(message):+ if not isinstance(message,dict):return ""+ content=message.get("content",[])+ if isinstance(content,str):return content+ if isinstance(content,list):+ return " ".join(x.get("text","") for x in content+ if isinstance(x,dict) and x.get("type")=="text")+ return ""++def file_prompt_matches(sid,prompt,journal_root=None,after=None):+ if not isinstance(sid,str) or not UUID.fullmatch(sid):return False+ root=Path(journal_root) if journal_root else Path.home()/".claude/projects"+ root=root.resolve()+ candidates=list(root.glob(f"*/{sid}.jsonl"))+ if len(candidates)!=1:return False+ p=candidates[0]+ if p.is_symlink() or not p.resolve().is_relative_to(root):return False+ expected=hashlib.sha256(prompt.encode()).digest()+ try:+ with p.open("rb") as f:+ f.seek(0,2);total=f.tell()+ f.seek(max(0,total-1200000))+ if total>1200000:f.readline()+ raw=f.readlines()+ except OSError:return False+ for line in reversed(raw):+ try:j=json.loads(line)+ except ValueError:continue+ if j.get("type")!="user":continue+ try:t=datetime.fromisoformat(j.get("timestamp","").replace("Z","+00:00")).timestamp()+ except (ValueError,TypeError):continue+ if after is not None and t < after-35:continue+ text=get_prompt(j.get("message"))+ if hashlib.sha256(text.encode()).digest()==expected:+ return True+ return False++def validate_handoff(agents,parent_id,run_id,job_id,prompt,started_at,expected_cwd,journal_root=None):+ errors=[]+ if not isinstance(agents,list):return {"verified":False,"errors":["agent list invalid"]}+ parents=[x for x in agents if isinstance(x,dict) and x.get("sessionId")==parent_id]+ children=[x for x in agents if isinstance(x,dict) and x.get("id")==job_id]+ if len(parents)!=1 or parents[0].get("state")!="stopped":+ errors.append("parent background session is not uniquely stopped")+ if len(children)!=1:+ errors.append("new job ID is missing or duplicated")+ child=None+ else:+ child=children[0]+ if child.get("sessionId")==parent_id:+ errors.append("not a new child")+ if not UUID.fullmatch(str(child.get("sessionId") or "")):+ errors.append("new session UUID invalid")+ if child.get("kind")!="background" or child.get("cwd")!=expected_cwd:+ errors.append("child kind or original working directory mismatch")+ if child.get("name")!="Recovery-"+run_id:+ errors.append("child name/target work-item mismatch")+ if child.get("status") not in ("busy","working","idle"):+ errors.append("child not an active background session")+ if not isinstance(child.get("startedAt"),(int,float)) or child.get("startedAt")/1000 < started_at-35:+ errors.append("child started before authorised launcher")+ busy=[x for x in agents if isinstance(x,dict) and+ x.get("kind")=="background" and x.get("status") in ("busy","working","running","active")]+ if child and any(x.get("sessionId")!=child.get("sessionId") for x in busy):+ errors.append("multiple background writers are active")+ prompt_evidence=False+ if child and not errors:+ prompt_evidence=file_prompt_matches(child["sessionId"],prompt,journal_root,started_at)+ if not prompt_evidence:errors.append("child journal lacks exact authorised prompt")+ return {+ "verified":not errors,+ "errors":errors,+ "parent_session":parent_id,+ "child_session":child.get("sessionId") if child else None,+ "child_job_id":job_id,+ "prompt_sha256":hashlib.sha256(prompt.encode()).hexdigest(),+ "prompt_in_child_journal":prompt_evidence,+ "exclusive_background_writer":len(busy)<=1,+ "may_publish":False,+ "may_mutate_cad":False,+ }++def atomic(path,raw):+ path=Path(path)+ path.parent.mkdir(parents=True,exist_ok=True,mode=0o700)+ tmp=path.with_name(path.name+".new-"+str(os.getpid()))+ try:+ fd=os.open(tmp,os.O_CREAT|os.O_EXCL|os.O_WRONLY,0o600)+ with os.fdopen(fd,"wb") as f:+ f.write(raw);f.flush();os.fsync(f.fileno())+ os.replace(tmp,path)+ finally:+ if tmp.exists():tmp.unlink()++def adopt_handoff(handoff,config_path,native_stop_path,state_root,run_id):+ """Privately transfer one existing run binding, preserving prior bindings.++ Takes an exclusive local lock. No external APIs, no edits to native PCB/run.+ """+ if not isinstance(handoff,dict) or handoff.get("verified") is not True:+ raise ValueError("unverified handoff not adoptable")+ config_path=Path(config_path)+ native_stop_path=Path(native_stop_path)+ state_root=Path(state_root)+ state_root.mkdir(parents=True,exist_ok=True,mode=0o700)+ lock=state_root/"handoff-exclusive.lock"+ with lock.open("a+") as f:+ fcntl.flock(f,fcntl.LOCK_EX|fcntl.LOCK_NB)+ before_config=config_path.read_bytes()+ before_stop=native_stop_path.read_bytes()+ cfg=json.loads(before_config)+ native=json.loads(before_stop)+ if (cfg.get("schemaVersion")!="1.0" or cfg.get("allow_external_submission") is not False+ or cfg.get("auto_dispatch") is not False):+ raise ValueError("unexpected recovery policy")+ matches=[x for x in cfg.get("runs",[]) if x.get("run_id")==run_id]+ if len(matches)!=1 or matches[0].get("session_id")!=handoff.get("parent_session"):+ raise ValueError("recovery ownership changed before handoff")+ record=matches[0]+ old=handoff["parent_session"]+ child=handoff["child_session"]+ if not UUID.fullmatch(str(child or "")):raise ValueError("bad child UUID")+ existing=[x for x in native.get("run_bindings",[]) if x.get("run_dir")==record["run_dir"]]+ if not existing:raise ValueError("no existing native AI Flow binding")+ source=existing[0]+ if any(x.get("session_id")==child for x in native["run_bindings"]):+ raise ValueError("duplicate native Stop-hook session binding")+ if source.get("ai_thread")!=json.loads((Path(record["run_dir"])/"run.json").read_text()).get("aiThread"):+ raise ValueError("native AI Flow thread differs from guard binding")+ new={**source,"session_id":child}+ native["run_bindings"].append(new)+ cfg["runs"]=[{**x,"session_id":child} if x.get("run_id")==run_id else x for x in cfg["runs"]]+ b1=state_root/("config-before-"+child+".json")+ b2=state_root/("native-stop-before-"+child+".json")+ if b1.exists() or b2.exists():raise ValueError("handoff already recorded")+ atomic(b1,before_config);atomic(b2,before_stop)+ if config_path.read_bytes()!=before_config or native_stop_path.read_bytes()!=before_stop:+ raise ValueError("concurrent configuration edit")+ next_stop=(json.dumps(native,indent=2)+"\n").encode()+ next_cfg=(json.dumps(cfg,indent=2)+"\n").encode()+ atomic(native_stop_path,next_stop)+ try:+ if config_path.read_bytes()!=before_config:raise ValueError("concurrent config edit")+ atomic(config_path,next_cfg)+ except Exception:+ if native_stop_path.read_bytes()==next_stop:+ atomic(native_stop_path,before_stop)+ raise+ receipt={**handoff,"run_id":run_id,"handoff_at":datetime.now(timezone.utc).isoformat(),+ "native_stop_hook_bound":True,"monitor_bound":True,+ "parent_session_preserved":True,"no_adom_submission":True}+ atomic(state_root/("handoff-"+child+".json"),(json.dumps(receipt,indent=2)+"\n").encode())+ return receipt+@@ -0,0 +1,228 @@+#!/usr/bin/env python3+import importlib.util+import json+from pathlib import Path+import tempfile+import unittest+from unittest.mock import patch+from types import SimpleNamespace++P=Path(__file__).resolve().parents[1]/"tools/dispatch.py"+spec=importlib.util.spec_from_file_location("recovery_dispatch",P)+m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m)+SID="12345678-1234-1234-1234-123456789abc"++def incident(**changes):+ return {"run_id":"RUNTIME1","stage":"placement","category":"idle_unfinished",+ "stalled_seconds":4200,"run_state":"shadow","new_progress":False,**changes}+def obs(**changes):+ return {"run_id":"RUNTIME1","stage":"placement","progress_key":"revision-one",+ "unfinished":True,"agent_status":"idle","active_operation":False,**changes}+def conf(**changes):+ return {"schemaVersion":"1.0","shadow_mode":False,"auto_dispatch":True,+ "allow_external_submission":False,"min_idle_seconds":360,**changes}+class DispatchTests(unittest.TestCase):+ def test_shadow_mode_never_dispatches(self):+ ok,reason=m.eligibility(incident(),obs(),conf(shadow_mode=True))+ self.assertFalse(ok);self.assertEqual(reason,"shadow_mode")+ def test_default_no_autodispatch(self):+ ok,reason=m.eligibility(incident(),obs(),conf(auto_dispatch=False))+ self.assertFalse(ok);self.assertEqual(reason,"auto_dispatch_disabled")+ def test_live_engineering_process_blocks_dispatch(self):+ ok,why=m.eligibility(incident(),obs(active_operation=True),conf())+ self.assertFalse(ok);self.assertEqual(why,"native_operation_active")+ def test_held_work_requires_owner(self):+ ok,why=m.eligibility(incident(),obs(owner_hold=True),conf())+ self.assertFalse(ok)+ def test_active_agent_blocks_duplicate(self):+ ok,why=m.eligibility(incident(),obs(agent_status="working"),conf())+ self.assertFalse(ok);self.assertEqual(why,"agent_not_idle")+ def test_no_unfinished_work_blocks(self):+ ok,why=m.eligibility(incident(),obs(unfinished=False),conf())+ self.assertFalse(ok)+ def test_below_stall_threshold(self):+ ok,why=m.eligibility(incident(stalled_seconds=20),obs(),conf())+ self.assertFalse(ok)+ def test_changed_revision_prevents_dispatch(self):+ ok,why=m.eligibility(incident(),obs(progress_key="rev-new"),conf(),+ {"candidate_key":"rev-old"})+ self.assertFalse(ok);self.assertEqual(why,"progress_changed")+ def test_hourly_attempt_budget_blocks_runaway_agent_spawning(self):+ past={"resume_timestamps":[9990,9980]}+ ok,reason=m.eligibility(incident(),obs(),conf(max_auto_resumes_per_hour=2),past,now=10000)+ self.assertFalse(ok);self.assertEqual(reason,"resume_hourly_budget_exhausted")++ def test_old_resume_attempts_expire_from_window(self):+ past={"resume_timestamps":[100,300]}+ ok,reason=m.eligibility(incident(),obs(),conf(max_auto_resumes_per_hour=2),past,now=10000)+ self.assertTrue(ok)++ def test_one_resume_per_revision(self):+ ok,why=m.eligibility(incident(),obs(),conf(),+ {"dispatched_for_key":"revision-one"})+ self.assertFalse(ok)+ def test_authorised_idle_run_eligible(self):+ ok,why=m.eligibility(incident(),obs(),conf())+ self.assertTrue(ok);self.assertEqual(why,"resume_existing_session_once")+ def test_external_submissions_disabled_even_with_config_allow_true(self):+ ok,why=m.eligibility(incident(),obs(),conf(allow_external_submission=True))+ self.assertFalse(ok)+ def test_normal_error_or_auth_never_dispatches(self):+ for kind in ("auth_required","tool_error","long_operation","loop","suspected_code_defect"):+ ok,why=m.eligibility(incident(category=kind),obs(),conf())+ self.assertFalse(ok)+ def test_doctor_safe_fix_disabled_by_default(self):+ ok,why=m.eligibility(incident(category="service_unhealthy"),obs(),conf())+ self.assertFalse(ok)+ def test_doctor_safe_fix_with_opt_in(self):+ ok,why=m.eligibility(incident(category="service_unhealthy"),obs(),+ conf(auto_safe_doctor_fix=True))+ self.assertTrue(ok);self.assertEqual(why,"doctor_safe_fix_once")+ def test_doctor_external_url_rejected(self):+ with self.assertRaises(ValueError):m.doctor_safe_fix("https://example.com/")+ with self.assertRaises(ValueError):m.doctor_safe_fix("http://10.0.0.1:2345")+ def test_doctor_safe_fix_verifies_before_and_after(self):+ from unittest.mock import MagicMock+ payloads=[+ {"before":{"failed":1,"checks":[{"check":"relay","ok":False}]}},+ {"actions":[{"name":"heal_relay"}],"recommend_workspace_restart":False},+ {"before":{"failed":0,"checks":[{"check":"relay","ok":True}]}}+ ]+ observed=[]+ def fake_urlopen(req,timeout):+ observed.append(req)+ body=json.dumps(payloads[len(observed)-1]).encode()+ response=MagicMock()+ response.read.return_value=body+ response.__enter__.return_value=response+ return response+ with patch.object(m,"urlopen",side_effect=fake_urlopen):+ result=m.doctor_safe_fix("http://127.0.0.1:4444")+ self.assertEqual(result["status"],"verified")+ self.assertEqual(result["before_failures"],1)+ self.assertEqual(result["after_failures"],0)+ self.assertEqual(len(observed),3)+ self.assertEqual(observed[1].get_method(),"POST")++ def test_doctor_no_failures_does_not_post(self):+ from unittest.mock import MagicMock+ response=MagicMock()+ response.read.return_value=json.dumps({"before":{"failed":0}}).encode()+ response.__enter__.return_value=response+ with patch.object(m,"urlopen",return_value=response) as http:+ r=m.doctor_safe_fix("http://localhost:2222")+ self.assertEqual(r["status"],"no_failures")+ self.assertEqual(http.call_count,1)++ def test_json_private_mode_and_no_secret_write(self):+ with tempfile.TemporaryDirectory() as folder:+ p=Path(folder)/"state.json"+ m.write(p,{"status":"shadow"})+ self.assertEqual(m.read(p)["status"],"shadow")+ self.assertEqual(p.stat().st_mode&0o777,0o600)+ def test_native_guard_rejects_owner_hold(self):+ proc=SimpleNamespace(returncode=0,stdout=json.dumps({+ "decision":"stop","mayStop":True,"currentStep":"placement"}))+ ok,reason=m.native_guard({"native_aiflow_binary":__file__,+ "run_dir":"/tmp/run"},runner=lambda *x,**k:proc)+ self.assertFalse(ok)+ def test_native_guard_requires_same_stage_and_no_stop_evidence(self):+ proc=SimpleNamespace(returncode=0,stdout=json.dumps({+ "decision":"continue","mayStop":False,"currentStep":"placement",+ "stopEvidence":None}))+ ok,stage=m.native_guard({"native_aiflow_binary":__file__,+ "run_dir":"/tmp/run"},runner=lambda *x,**k:proc)+ self.assertTrue(ok);self.assertEqual(stage,"placement")+ proc.stdout=json.dumps({"decision":"continue","mayStop":False,+ "currentStep":"placement","stopEvidence":{"owner":"hold"}})+ self.assertFalse(m.native_guard({"native_aiflow_binary":__file__,+ "run_dir":"/tmp/run"},runner=lambda *x,**k:proc)[0])+ def fixture(self,base):+ root=Path(base);rid="RUNTIME1";run=root/"run";run.mkdir()+ (run/"run.json").write_text('{"currentStep":"placement"}')+ state=root/"state";node=state/rid;node.mkdir(parents=True)+ (node/"latest.json").write_text(json.dumps(incident()))+ config=root/"config.json"+ config.write_text(json.dumps({**conf(),"runs":[{+ "run_id":rid,"run_dir":str(run),"session_id":SID,+ "native_aiflow_binary":__file__,"working_directory":str(root),+ "agent_name":"RecoveryAgent"}]}))+ return config,state,node+ def test_dry_run_does_not_spawn_or_write_ledger(self):+ with tempfile.TemporaryDirectory() as d:+ cfg,root,node=self.fixture(d)+ with patch.object(m,"native_guard",side_effect=AssertionError("no native gate on dry-run")),patch.object(m,"agents",side_effect=AssertionError("no dispatch")):+ c=m.read(cfg);c["shadow_mode"]=True;m.write(cfg,c)+ out=m.run_once(cfg,root,apply=False)+ self.assertEqual(out[0]["status"],"skip")+ self.assertFalse((node/"dispatch-ledger.json").exists())+ def test_one_real_dispatch_is_idempotent_under_fake_claude(self):+ with tempfile.TemporaryDirectory() as d:+ cfg,root,node=self.fixture(d)+ calls=[]+ def fake_runner(cmd,**kw):+ calls.append(cmd)+ return SimpleNamespace(returncode=0,stdout="agent launched",stderr="")+ agent=[{"sessionId":SID,"name":"RecoveryAgent","status":"idle"}]+ result=m.run_once(cfg,root,apply=True,runner=fake_runner,+ collector=lambda x:obs(),+ agent_lister=lambda:agent,+ guard=lambda run,runner:(True,"placement"))+ self.assertEqual(result[0]["status"],"resume_invoked")+ self.assertEqual(len(calls),1)+ self.assertIn("--resume",calls[0])+ self.assertIn(SID,calls[0])+ text=(node/"dispatch-ledger.json").read_text()+ self.assertNotIn("agent launched",text)+ again=m.run_once(cfg,root,apply=True,runner=fake_runner,+ collector=lambda x:obs(),agent_lister=lambda:agent,+ guard=lambda run,runner:(True,"placement"))+ self.assertEqual(again[0]["status"],"skip")+ self.assertEqual(len(calls),1)++ def test_native_guard_refusal_is_never_bypassed(self):+ with tempfile.TemporaryDirectory() as d:+ cfg,root,node=self.fixture(d)+ done=m.run_once(cfg,root,apply=True,+ collector=lambda x:obs(),+ guard=lambda run,runner:(False,"owner_stop"))+ self.assertEqual(done[0]["status"],"refused_native_guard")+ self.assertEqual(m.read(node/"dispatch-ledger.json")["status"],+ "refused_native_guard")++ def test_duplicate_live_agent_refused(self):+ with tempfile.TemporaryDirectory() as d:+ cfg,root,node=self.fixture(d)+ result=m.run_once(cfg,root,apply=True,+ collector=lambda x:obs(),+ guard=lambda run,runner:(True,"placement"),+ agent_lister=lambda:[+ {"sessionId":SID,"name":"RecoveryAgent","status":"idle"},+ {"sessionId":"other","name":"RecoveryAgent","status":"working"}])+ self.assertEqual(result[0]["status"],"refused_competing_agent")++ def test_false_native_guard_stage_refused(self):+ with tempfile.TemporaryDirectory() as d:+ cfg,root,node=self.fixture(d)+ result=m.run_once(cfg,root,apply=True,+ collector=lambda x:obs(),+ guard=lambda run,runner:(True,"routing"))+ self.assertEqual(result[0]["status"],"refused_native_guard")++ def test_no_pr_upstream_codepaths(self):+ contents=P.read_text()+ for disallowed in ("adom-wiki issue create","adom-wiki pr create",+ "repo push","git push","git clone"):+ self.assertNotIn(disallowed,contents)+ def test_stale_run_incident_refused(self):+ with tempfile.TemporaryDirectory() as d:+ cfg,root,node=self.fixture(d)+ i=m.read(node/"latest.json");i["stage"]="routing";m.write(node/"latest.json",i)+ out=m.run_once(cfg,root,apply=True)+ self.assertEqual(out[0]["status"],"skip")+ self.assertEqual(out[0]["reason"],"stale_incident")+ self.assertFalse((node/"dispatch-ledger.json").exists())++if __name__=="__main__":+ unittest.main(verbosity=2)+@@ -0,0 +1,265 @@+#!/usr/bin/env python3+import importlib.util+import json+from pathlib import Path+import tempfile+import unittest+from unittest.mock import patch+from contextlib import redirect_stdout+import io++SOURCE=Path(__file__).resolve().parents[1] / "tools/triage.py"+spec=importlib.util.spec_from_file_location("recovery",SOURCE)+r=importlib.util.module_from_spec(spec)+spec.loader.exec_module(r)+BASE={"run_id":"test-run","stage":"routing","progress_key":"rev1","unfinished":True,+ "agent_status":"working","declared_state":"working",+ "active_operation":False,"required_tools":[],"doctor_checks":[]}++def obs(**kwargs):return {**BASE,**kwargs}++class RecoveryTests(unittest.TestCase):+ def assertCategory(self,expected,o,previous=None,now=10000,policy=None):+ result,state=r.evaluate(o,previous,policy,now)+ self.assertEqual(result["category"],expected,result)+ self.assertFalse(result["may_publish"])+ self.assertFalse(result["may_spawn_agent"])+ self.assertFalse(result["may_relax_gates"])+ return result,state++ def test_working(self):self.assertCategory("working",obs(),{"progress_key":"rev1","progress_since":9990})+ def test_new_progress_resets_clock(self):+ result,state=self.assertCategory("working",obs(progress_key="rev2"),{"progress_key":"rev1","progress_since":100})+ self.assertTrue(result["new_progress"]);self.assertEqual(state["progress_since"],10000)+ def test_idle_when_progress_stale(self):+ a,s=self.assertCategory("idle_unfinished",obs(agent_status="idle"),+ {"progress_key":"rev1","progress_since":9500})+ self.assertTrue(a["alert"])+ def test_idle_below_threshold(self):+ self.assertCategory("healthy",obs(agent_status="idle"),+ {"progress_key":"rev1","progress_since":9750})+ def test_long_tool_not_assumed_idle(self):+ self.assertCategory("long_operation",obs(active_operation=True,agent_status="idle"),+ {"progress_key":"rev1","progress_since":8000})+ def test_normal_long_operation(self):+ self.assertCategory("working",obs(active_operation=True),+ {"progress_key":"rev1","progress_since":9990})+ def test_busy_agent_with_recent_tool_activity_is_working(self):+ self.assertCategory("working",+ obs(agent_status="busy",recent_tool_at=9950),+ {"progress_key":"rev1","progress_since":1000})+ def test_busy_agent_without_recent_tool_event_is_not_progress(self):+ self.assertCategory("no_progress",+ obs(agent_status="busy",recent_tool_at=8000),+ {"progress_key":"rev1","progress_since":1000})+ def test_no_progress(self):+ self.assertCategory("no_progress",obs(agent_status="unknown",declared_state=""),+ {"progress_key":"rev1","progress_since":9000})+ def test_completed(self):+ self.assertCategory("completed",obs(completed=True,unfinished=False),+ {"progress_key":"rev1","progress_since":100})+ def test_maintenance_hold(self):+ self.assertCategory("paused",obs(maintenance=True,agent_status="idle"),+ {"progress_key":"rev1","progress_since":100})+ def test_user_hold(self):+ self.assertCategory("owner_decision",obs(owner_hold=True),None)+ def test_authentication_never_restarts(self):+ x,_=self.assertCategory("auth_required",obs(event={"code":"401 token expired"}))+ self.assertEqual(x["route"],"owner")+ self.assertFalse(x["safe_autoheal_eligible"])+ def test_missing_required_binary(self):+ x,_=self.assertCategory("missing_required_app",+ obs(required_tools=[{"name":"KiCad CLI","available":False,"required":True},+ {"name":"optional FFmpeg","available":False,"required":False}]))+ self.assertEqual(x["evidence"],["KiCad CLI"])+ def test_optional_tool_absence_not_blocker(self):+ self.assertCategory("working",obs(required_tools=[{"name":"FFmpeg","available":False,"required":False}]))+ def test_missing_program_in_tool_error(self):+ a,_=self.assertCategory("missing_required_app",+ obs(event={"code":"adom-cli: command not found"}))+ self.assertEqual(a["route"],"environment")++ def test_misconfigured(self):+ self.assertCategory("misconfigured_dependency",obs(event={"code":"missing environment variable"}))+ def test_service_unhealthy(self):+ x,_=self.assertCategory("service_unhealthy",+ obs(doctor_checks=[{"id":"relay","ok":False,"required":True}]))+ self.assertTrue(x["safe_autoheal_eligible"])+ def test_service_unhealthy_during_route_never_autoheals(self):+ x,_=self.assertCategory("service_unhealthy",+ obs(active_operation=True,doctor_checks=[{"id":"relay","ok":False,"required":True}]))+ self.assertFalse(x["safe_autoheal_eligible"])+ def test_ignored_doctor_check(self):+ self.assertCategory("working",obs(doctor_checks=[{"id":"cosmetic","ok":False,"required":False}]))+ def test_tool_error_before_repeat(self):+ x,s=self.assertCategory("tool_error",obs(event={"code":"camera ffmpeg error"}))+ self.assertEqual(s["repeats"],1)+ def test_repeat_error_becomes_loop(self):+ failure=obs(event={"code":"same failure"})+ x,s=r.evaluate(failure,None,now=9000)+ x,s=r.evaluate(failure,s,now=9250)+ x,s=r.evaluate(failure,s,now=9500)+ self.assertEqual(x["category"],"loop")+ self.assertEqual(x["same_failure_repeats"],3)+ def test_repeated_sampling_of_same_tool_event_does_not_make_a_loop(self):+ v=obs(event={"code":"same failure","event_id":"tool-result-A"})+ _,state=r.evaluate(v,None,now=1000)+ _,state=r.evaluate(v,state,now=1200)+ latest,state=r.evaluate(v,state,now=1400)+ self.assertEqual(latest["same_failure_repeats"],1)+ self.assertEqual(latest["category"],"tool_error")++ def test_distinct_tool_failures_can_form_a_loop(self):+ s=None+ for i in range(3):+ latest,s=r.evaluate(obs(event={"code":"same failure","event_id":f"tool-{i}"}),s,now=1000+200*i)+ self.assertEqual(latest["category"],"loop")++ def test_progress_resets_loop(self):+ failure=obs(event={"code":"same failure"})+ _,s=r.evaluate(failure,None,now=9000)+ _,s=r.evaluate(failure,s,now=9250)+ a,s=r.evaluate({**failure,"progress_key":"rev2"},s,now=9500)+ self.assertEqual(a["same_failure_repeats"],1)+ self.assertEqual(a["category"],"tool_error")+ def test_reproduced_source_defect_requires_fixture(self):+ self.assertCategory("tool_error",obs(event={"code":"bug","reproduced_defect":True}))+ a,_=self.assertCategory("suspected_code_defect",+ obs(event={"code":"bug","reproduced_defect":True,"isolated_fixture":True}))+ self.assertEqual(a["route"],"assurance")+ def test_cooldown(self):+ o=obs(event={"code":"same failure"})+ a,s=r.evaluate(o,None,now=10000)+ b,s=r.evaluate(o,s,now=10020)+ self.assertFalse(b["alert"])+ self.assertTrue(b["suppressed_by_cooldown"])+ def test_fresh_error_not_suppressed(self):+ a,s=r.evaluate(obs(event={"code":"different A"}),None,now=10000)+ b,s=r.evaluate(obs(event={"code":"different B"}),s,now=10020)+ self.assertTrue(b["alert"])+ def test_different_stage_not_same_fingerprint(self):+ a,s=r.evaluate(obs(event={"code":"same failure"}),None,now=10000)+ b,s=r.evaluate(obs(stage="placement",event={"code":"same failure"}),s,now=10020)+ self.assertEqual(b["same_failure_repeats"],1)+ def test_missing_progress_key_fails(self):+ with self.assertRaises(ValueError):r.evaluate(obs(progress_key=""))+ def test_policy_rejects_bad_settings(self):+ with self.assertRaises(ValueError):r.evaluate(obs(),policy={"loop_repeats":0})+ with self.assertRaises(ValueError):r.evaluate(obs(),policy={"idle_seconds":"zero"})+ def test_all_decisions_are_known(self):+ for name in r.CATEGORIES:+ self.assertIn(name,r.REMEDIATION)+ def test_json_file_not_path_escape(self):+ with tempfile.TemporaryDirectory() as t:+ p=Path(t)/"state.json"+ r.atomic_json(p,{"ok":True})+ self.assertEqual(r.json_file(p,{}),{"ok":True})+ self.assertEqual(p.stat().st_mode&0o777,0o600)+ def test_doctor_url_rejects_external_address(self):+ self.assertEqual(r.capture_doctor({"doctor_enabled":True,"doctor_url":"https://example.com"})[0]["kind"],"configuration")+ def test_doctor_disabled_no_request(self):+ self.assertEqual(r.capture_doctor({"doctor_enabled":False}),[])+ def test_config_requires_uuid_and_safe_run(self):+ c={"schemaVersion":"1.0","runs":[{"run_id":"RUN1","run_dir":"/tmp/dir/run","session_id":"a"*8}]}+ with self.assertRaises(ValueError):r.validate_config(c)+ c["runs"][0]["session_id"]="12345678-1234-1234-1234-123456789abc"+ self.assertEqual(r.validate_config(c),c)+ def test_hook_ignores_unknown_session(self):+ with tempfile.TemporaryDirectory() as t:+ root=Path(t)+ config=root/"config.json"+ config.write_text(json.dumps({"schemaVersion":"1.0","runs":[{+ "run_id":"case_one","run_dir":str(root/"run"),+ "session_id":"12345678-1234-1234-1234-123456789abc"}]}))+ rc=r.action_hook(config,root,json.dumps({+ "session_id":"eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee",+ "hook_event_name":"Stop"}))+ self.assertEqual(rc,0)+ self.assertFalse((root/"case_one/last-hook.json").exists())+ def test_hook_records_only_whitelisted_fields(self):+ with tempfile.TemporaryDirectory() as t:+ root=Path(t);config=root/"config.json"+ sid="12345678-1234-1234-1234-123456789abc"+ config.write_text(json.dumps({"schemaVersion":"1.0","runs":[{+ "run_id":"case_one","run_dir":str(root/"run"),"session_id":sid}]}))+ r.action_hook(config,root,json.dumps({"hook_event_name":"Stop",+ "session_id":sid,"transcript_path":"/private/user/cad","api_key":"DONT_STORE"}))+ payload=(root/"case_one/last-hook.json").read_text()+ self.assertNotIn("DONT_STORE",payload)+ self.assertNotIn("cad",payload)+ self.assertIn('"no_dispatch": true',payload)+ def test_3000_independent_work_items_and_25_stage_labels(self):+ for i in range(3000):+ a,_=r.evaluate(obs(run_id=f"BOARD-{i}",stage=f"stage-{i%25+1}"),now=1000)+ self.assertTrue(a["run_id"].startswith("BOARD-"))+ def test_timeline_server_does_not_count_as_engineering_work(self):+ ps="""adom-aiflow /opt/adom-aiflow --run /workspace/run timeline serve --port 8894+python3 /usr/bin/python3 /opt/hsc01-standby-server.py 8894 /opt/adom-aiflow --run /workspace/run timeline serve+bash /bin/bash -lc 'adom-aiflow --run /workspace/run route --json'+"""+ self.assertEqual(r.identify_active_engineering_processes(+ ps,["adom-aiflow --run","kicad_route_net"]),[])++ def test_real_native_routing_process_counts(self):+ ps="""adom-aiflow /opt/adom-aiflow --run /workspace/run route --passes 4+python3 python3 kicad_route_net.py --board fixture.kicad_pcb+java java -jar freerouting.jar fixture.dsn+"""+ self.assertEqual(r.identify_active_engineering_processes(+ ps,["adom-aiflow --run","kicad_route_net"]),["adom-aiflow --run","kicad_route_net"])++ def test_non_router_python_does_not_count(self):+ ps="""python3 /usr/bin/python3 /opt/web-server.py --port 8894+bash /bin/bash -lc 'kicad_route_net.py'+"""+ self.assertEqual(r.identify_active_engineering_processes(+ ps,["kicad_route_net"]),[])++ def test_live_watch_requires_explicit_run_binding(self):+ with tempfile.TemporaryDirectory() as t:+ root=Path(t)+ config=root/"config.json"+ config.write_text(json.dumps({"schemaVersion":"1.0","runs":[]}))+ with self.assertRaises(ValueError):r.action_watch(config,root/"output")+ def test_collect_fixture_run(self):+ with tempfile.TemporaryDirectory() as t:+ root=Path(t);run=root/"run";run.mkdir()+ (run/"run.json").write_text(json.dumps({"currentStep":"placement","status":"running"}))+ with patch.object(r.subprocess,"run") as subprocess_run:+ subprocess_run.return_value.returncode=0+ subprocess_run.return_value.stdout="[]"+ o=r.collect({"run_id":"FIXTURE","run_dir":str(run),"session_id":"12345678-1234-1234-1234-123456789abc",+ "required_tools":[{"name":"nonexistent-test-program-xyzz","required":True}]})+ self.assertEqual(o["stage"],"placement")+ self.assertTrue(o["unfinished"])+ self.assertFalse(o["required_tools"][0]["available"])+ self.assertTrue(o["progress_key"])+ def test_shadow_commission_creates_private_evidence(self):+ with tempfile.TemporaryDirectory() as t:+ root=Path(t);run=root/"run";run.mkdir()+ (run/"run.json").write_text(json.dumps({"currentStep":"routing"}))+ conf=root/"config.json"+ sid="12345678-1234-1234-1234-123456789abc"+ conf.write_text(json.dumps({"schemaVersion":"1.0","shadow_mode":True,+ "runs":[{"run_id":"HSC","run_dir":str(run),"session_id":sid,"required_tools":[]}]}))+ with patch.object(r,"collect",return_value=obs(run_id="HSC")):+ v=r.action_watch(conf,root/"state",now=1000)+ self.assertEqual(len(v),1)+ self.assertEqual(v[0]["run_state"],"shadow")+ self.assertTrue((root/"state/HSC/latest.json").exists())+ self.assertFalse((root/"state/HSC/contribution-assurance-handoff.json").exists())+ self.assertEqual(list((root/"state/HSC").glob("*.prompt")),[])+ def test_initial_unchanged_run_uses_existing_board_timestamp(self):+ with tempfile.TemporaryDirectory() as t:+ root=Path(t);run=root/"run";run.mkdir()+ (run/"run.json").write_text('{"currentStep":"routing"}')+ conf=root/"config.json";sid="12345678-1234-1234-1234-123456789abc"+ conf.write_text(json.dumps({"schemaVersion":"1.0","runs":[{+ "run_id":"ONE","run_dir":str(run),"session_id":sid}]}))+ with patch.object(r,"collect",return_value=obs(run_id="ONE",agent_status="idle",initial_progress_since=100)):+ v=r.action_watch(conf,root/"state",now=1000)+ self.assertEqual(v[0]["category"],"idle_unfinished")++if __name__=="__main__":+ unittest.main(verbosity=2)+@@ -0,0 +1,216 @@+#!/usr/bin/env python3+"""Isolated recovery execution gate tests; NEVER launches Claude."""+import importlib.util+import json+import os+from pathlib import Path+import tempfile+import unittest+from unittest.mock import patch+from datetime import datetime, timezone+import subprocess++file=Path(__file__).resolve().parents[1] / "tools/recover.py"+spec=importlib.util.spec_from_file_location("gate",file)+g=importlib.util.module_from_spec(spec)+spec.loader.exec_module(g)+SID="e8da74b2-236b-4ecd-8b5e-a0f2b9d7ba29"+def a(status="idle",sid=SID,kind="background"):+ return {"sessionId":sid,"id":"e8da74b2","status":status,"kind":kind,"state":"working"}++class RecoveryGateTests(unittest.TestCase):+ def setUp(self):+ self.temp=tempfile.TemporaryDirectory()+ self.addCleanup(self.temp.cleanup)+ self.root=Path(self.temp.name)+ self.run=self.root/"case"/"run";self.run.mkdir(parents=True)+ self.board=self.root/"board.kicad_pcb"+ self.board.write_text("(kicad_pcb)")+ self.file=self.run/"run.json"+ self.file.write_text(json.dumps({"currentStep":"placement",+ "sequence":["intake","placement","routing","complete"],"status":"working"}))+ os.utime(self.board,(9500,9500));os.utime(self.file,(9500,9500))+ self.cfg={"schemaVersion":"1.0","allow_external_submission":False,+ "auto_dispatch":False,+ "runs":[{"run_id":"fixture","run_dir":str(self.run),+ "board_path":str(self.board),"session_id":SID}]}+ self.state=self.root/"state"+ (self.state/"fixture").mkdir(parents=True)+ (self.state/"fixture/latest.json").write_text(json.dumps({+ "category":"idle_unfinished","stage":"placement","run_state":"shadow",+ "timestamp":datetime.fromtimestamp(9995,timezone.utc).isoformat(),+ "may_publish":False,"may_spawn_agent":False,+ }))+ skill=self.root/".claude/skills/hydrogen-runtime-recovery/tools/triage.py"+ skill.parent.mkdir(parents=True)+ skill.write_text('def collect(config):\n return {"active_operation":False,"active_process_markers":[],"progress_key":"fixture"}\ndef capture_doctor(config):\n return [{"id":"test","ok":True} for _ in range(19)]\n')+ patches=[+ patch.object(g,"STATE",self.state),+ patch.object(Path,"home",return_value=self.root),+ patch.object(g,"collect_agents",return_value=[a()]),+ patch.object(g.time,"time",return_value=10000),+ ]+ for item in patches:+ item.start();self.addCleanup(item.stop)+ def check(self):+ return g.attest(self.cfg,"fixture",now=10000)+ def test_genuine_idle_verified_for_resume(self):+ report=self.check()+ self.assertTrue(report["approved"],report["reasons"])+ self.assertEqual(report["permitted_next_action"],"resume_same_session")+ self.assertEqual(report["previous_attempts_at_revision"],0)+ def test_brief_must_be_inside_owning_project(self):+ self.cfg["runs"][0]["work_item_brief"]="/etc/passwd"+ self.assertIn("configured work item brief missing or outside owning project",self.check()["reasons"])++ def test_same_engine_supports_multiple_board_names(self):+ self.cfg["runs"][0]["run_id"]="GENERIC_R54"+ self.assertTrue(g.attest(self.cfg,"GENERIC_R54",now=10000)["approved"] is False)+ # The old observer is still bound to fixture, so it MUST not silently+ # move an engineering authority to a differently named work item.++ def test_mismatched_original_cwd_blocks_configured_binding(self):+ self.cfg["runs"][0]["session_cwd"]="/tmp"+ self.assertIn("configured session working directory differs from the original agent",+ self.check()["reasons"])++ def test_wrong_run_cannot_resume(self):+ self.assertFalse(g.attest(self.cfg,"another",now=10000)["approved"])+ def test_missing_report_blocks(self):+ (self.state/"fixture/latest.json").unlink()+ self.assertFalse(self.check()["approved"])+ def test_report_current_stage_must_match_native(self):+ p=self.state/"fixture/latest.json"+ o=json.loads(p.read_text());o["stage"]="routing";p.write_text(json.dumps(o))+ self.assertFalse(self.check()["approved"])+ def test_recent_native_change_blocks(self):+ os.utime(self.board,(9975,9975))+ self.assertIn("board updated too recently to treat as stuck",self.check()["reasons"])+ def test_completed_board_blocks(self):+ self.file.write_text(json.dumps({"currentStep":"complete",+ "sequence":["intake","placement","routing","complete"],"status":"completed"}))+ self.assertFalse(self.check()["approved"])+ def test_competing_agent_blocks(self):+ with patch.object(g,"collect_agents",return_value=[a(),a("working","11111111-1111-1111-1111-111111111111")]):+ self.assertFalse(self.check()["approved"])+ def test_bound_agent_non_idle_blocks(self):+ with patch.object(g,"collect_agents",return_value=[a("working")]):+ self.assertFalse(self.check()["approved"])+ def test_missing_doctor_blocks(self):+ t=self.root/".claude/skills/hydrogen-runtime-recovery/tools/triage.py"+ t.write_text('def collect(config):\n return {"active_operation":False,"active_process_markers":[],"progress_key":"x"}\ndef capture_doctor(config):\n return []\n')+ self.assertFalse(self.check()["approved"])+ def test_snapshot_hash_changes_when_board_changes(self):+ one=self.check()+ self.board.write_text("(kicad_pcb changed)")+ os.utime(self.board,(9500,9500))+ two=self.check()+ self.assertNotEqual(one["revision_fingerprint"],two["revision_fingerprint"])+ def test_retry_budget(self):+ original=self.check()+ p=self.state/"fixture/recovery-attempts.json"+ p.write_text(json.dumps({"attempts":[+ {"fingerprint":original["revision_fingerprint"],"time":9000,"status":"launch_requested"},+ {"fingerprint":original["revision_fingerprint"],"time":9300,"status":"launch_requested"}+ ]}))+ report=self.check()+ self.assertIn("retry budget exhausted for unchanged native revision",report["reasons"])+ def test_cooldown(self):+ original=self.check()+ p=self.state/"fixture/recovery-attempts.json"+ p.write_text(json.dumps({"attempts":[{"fingerprint":original["revision_fingerprint"],"time":9800,"status":"launch_requested"}]}))+ self.assertIn("recovery cooldown has not elapsed",self.check()["reasons"])+ def test_observer_cannot_authorise_publication(self):+ p=self.state/"fixture/latest.json";o=json.loads(p.read_text())+ o["may_publish"]=True;p.write_text(json.dumps(o))+ self.assertFalse(self.check()["approved"])+ def test_stale_plan_prevents_executor_calls(self):+ plan=self.check()+ plan["native_board_sha256"]="0"*64+ path=self.root/"plan.json";path.write_text(json.dumps(plan))+ with patch.object(g,"call") as commands:+ result=g.execute(self.cfg,"fixture",str(path))+ self.assertFalse(result["executed"])+ self.assertEqual(result["reason"],"stale_approved_plan")+ commands.assert_not_called()+ def test_no_progress_with_confirmed_stopped_bound_session(self):+ p=self.state/"fixture/latest.json"+ report=json.loads(p.read_text())+ report["category"]="no_progress"+ p.write_text(json.dumps(report))+ stopped={**a(),"state":"stopped","status":None}+ with patch.object(g,"collect_agents",side_effect=[[],[stopped]]):+ result=self.check()+ self.assertTrue(result["approved"],result["reasons"])++ def test_no_progress_with_ordinary_idle_agent_is_not_enough(self):+ p=self.state/"fixture/latest.json"+ report=json.loads(p.read_text())+ report["category"]="no_progress"+ p.write_text(json.dumps(report))+ self.assertIn("no_progress recovery requires the bound background session already stopped",+ self.check()["reasons"])++ def test_stopped_session_safe_to_resume_without_second_stop(self):+ stopped={**a(),"state":"stopped","status":None}+ with patch.object(g,"collect_agents",side_effect=[[],[stopped]]):+ plan=self.check()+ self.assertTrue(plan["approved"],plan["reasons"])+ self.assertTrue(plan["bound_session_stopped"])+ path=self.root/"plan.json";path.write_text(json.dumps(plan))+ actions=[]+ def fake(*args,**kwargs):+ actions.append(args)+ return subprocess.CompletedProcess(args,0,"e8da74b2","")+ with patch.object(g,"collect_agents",side_effect=[+ [],[stopped],[],[a("working")]]),patch.object(g,"call",side_effect=fake):+ outcome=g.execute(self.cfg,"fixture",str(path))+ self.assertTrue(outcome["executed"],outcome)+ self.assertEqual(len(actions),1)+ self.assertEqual(actions[0][:2],("claude","--bg"))++ def test_successful_mock_resume_uses_exact_same_id(self):+ plan=self.check();path=self.root/"plan.json";path.write_text(json.dumps(plan))+ calls=[]+ def mocked(*args,**kwargs):+ calls.append(args)+ if args[:2]==("claude","stop"):+ self.assertEqual(args[2],"e8da74b2")+ return subprocess.CompletedProcess(args,0,"stopped","")+ if args[:2]==("claude","--bg"):+ self.assertIn("--resume",args)+ self.assertEqual(args[args.index("--resume")+1],SID)+ self.assertNotIn("--fork-session",args)+ return subprocess.CompletedProcess(args,0,"e8da74b2","")+ raise AssertionError(args)+ with patch.object(g,"collect_agents",side_effect=[+ [a("idle")],[a("idle")],[a("stopped")],[a("working")]]),patch.object(g,"call",side_effect=mocked):+ result=g.execute(self.cfg,"fixture",str(path))+ self.assertTrue(result["executed"],result)+ self.assertFalse(result["engineering_gate_passed"])+ self.assertEqual(len(calls),2)+ ledger=json.loads((self.state/"fixture/recovery-attempts.json").read_text())+ self.assertEqual(ledger["attempts"][0]["status"],"launch_requested")+ self.assertEqual(ledger["attempts"][0]["session_id"],SID)+ def test_mock_stop_failure_never_attempts_resume(self):+ plan=self.check();path=self.root/"plan.json";path.write_text(json.dumps(plan))+ cmds=[]+ def mocked(*args,**kwargs):+ cmds.append(args)+ return subprocess.CompletedProcess(args,2,"","cannot stop")+ with patch.object(g,"call",side_effect=mocked):+ result=g.execute(self.cfg,"fixture",str(path))+ self.assertEqual(result["reason"],"claude_stop_failed")+ self.assertEqual(len(cmds),1)+ def test_lock_file_is_created_under_private_state(self):+ plan=self.check();path=self.root/"plan.json";path.write_text(json.dumps(plan))+ with patch.object(g,"call",return_value=subprocess.CompletedProcess([],2,"","blocked")):+ g.execute(self.cfg,"fixture",str(path))+ self.assertTrue((self.state/"fixture/exclusive-resume.lock").exists())+ def test_publication_config_blocks(self):+ self.cfg["allow_external_submission"]=True+ with self.assertRaises(RuntimeError):self.check()++if __name__=="__main__":+ unittest.main(verbosity=2)+@@ -0,0 +1,109 @@+#!/usr/bin/env python3+from pathlib import Path+import importlib.util+from datetime import datetime,timezone+import json,os,tempfile,unittest,time+from unittest.mock import patch++P=Path(__file__).resolve().parents[1]/"tools/handoff.py"+spec=importlib.util.spec_from_file_location("handoff",P)+h=importlib.util.module_from_spec(spec);spec.loader.exec_module(h)+OLD="e8da74b2-236b-4ecd-8b5e-a0f2b9d7ba29"+NEW="f530e41e-fe6f-4e92-93dd-2784061cabd0"+START=10000+PROMPT="Continue original existing authorised native engineering run with frozen limits."++class HandoffTests(unittest.TestCase):+ def setUp(self):+ self.tmp=tempfile.TemporaryDirectory();self.addCleanup(self.tmp.cleanup)+ self.root=Path(self.tmp.name)+ self.jroot=self.root/"journals";(self.jroot/"project").mkdir(parents=True)+ self.path=self.jroot/"project"/(NEW+".jsonl")+ self.path.write_text(json.dumps({"timestamp":datetime.fromtimestamp(START+4,timezone.utc).isoformat(),+ "type":"user","message":{"content":PROMPT}})+"\n")+ self.agents=[+ {"id":"e8da74b2","sessionId":OLD,"kind":"background","state":"stopped","cwd":"/tmp","name":"Old"},+ {"id":"f530e41e","sessionId":NEW,"kind":"background","status":"busy","state":"working",+ "cwd":"/tmp","name":"Recovery-fixture","startedAt":(START+2)*1000},+ ]+ def evaluate(self,agents=None,prompt=PROMPT):+ return h.validate_handoff(agents or self.agents,OLD,"fixture","f530e41e",+ prompt,START,"/tmp",self.jroot)+ def test_parse_launcher_short_id(self):+ self.assertEqual(h.launch_job_id("backgrounded · f530e41e · Recovery-fixture"),"f530e41e")+ def test_reject_bad_launcher_response(self):+ self.assertIsNone(h.launch_job_id("background failed because no connectivity"))+ def test_prompt_string_and_block_format(self):+ self.assertEqual(h.get_prompt({"content":PROMPT}),PROMPT)+ self.assertEqual(h.get_prompt({"content":[{"type":"text","text":PROMPT}]}),PROMPT)+ def test_exact_prompt_verified(self):+ o=self.evaluate()+ self.assertTrue(o["verified"],o["errors"])+ self.assertTrue(o["prompt_in_child_journal"])+ self.assertFalse(o["may_publish"])+ def test_mismatched_prompt_blocks(self):+ o=self.evaluate(prompt=PROMPT+" DIFFERENT")+ self.assertFalse(o["verified"])+ def test_old_still_active_blocks(self):+ agents=json.loads(json.dumps(self.agents));agents[0]["state"]="working"+ self.assertFalse(self.evaluate(agents)["verified"])+ def test_wrong_cwd_blocks(self):+ agents=json.loads(json.dumps(self.agents));agents[1]["cwd"]="/other"+ self.assertFalse(self.evaluate(agents)["verified"])+ def test_other_writer_blocks(self):+ agents=json.loads(json.dumps(self.agents))+ agents.append({"id":"abcdef12","sessionId":"faaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",+ "kind":"background","status":"busy"})+ self.assertFalse(self.evaluate(agents)["verified"])+ def test_old_child_session_not_acceptable(self):+ agents=json.loads(json.dumps(self.agents));agents[1]["sessionId"]=OLD+ self.assertFalse(self.evaluate(agents)["verified"])+ def test_child_started_too_early(self):+ agents=json.loads(json.dumps(self.agents));agents[1]["startedAt"]=(START-100)*1000+ self.assertFalse(self.evaluate(agents)["verified"])+ def test_wrong_child_name(self):+ agents=json.loads(json.dumps(self.agents));agents[1]["name"]="Some-Unrelated-Project"+ self.assertFalse(self.evaluate(agents)["verified"])+ def test_stale_prompt_timestamp(self):+ self.path.write_text(json.dumps({"timestamp":datetime.fromtimestamp(START-2000,timezone.utc).isoformat(),+ "type":"user","message":{"content":PROMPT}})+"\n")+ self.assertFalse(self.evaluate()["verified"])+ def test_journal_must_be_under_expected_directory(self):+ self.path.unlink()+ self.assertFalse(self.evaluate()["verified"])+ def write_config(self):+ run=self.root/"owner/board/run";run.mkdir(parents=True)+ (run/"run.json").write_text(json.dumps({"aiThread":"test-thread","currentStep":"placement","sequence":["placement","routing"]}))+ cfg=self.root/"config.json"+ cfg.write_text(json.dumps({"schemaVersion":"1.0","allow_external_submission":False,"auto_dispatch":False,+ "runs":[{"run_id":"fixture","run_dir":str(run),"session_id":OLD,"session_cwd":"/tmp"}]}))+ native=self.root/"native-stop.json"+ native.write_text(json.dumps({"run_bindings":[{"session_id":"aaaa0000-0000-0000-0000-000000000000",+ "board":"BOARDX","ai_thread":"test-thread","run_dir":str(run)}]}))+ return cfg,native+ def test_adopt_writes_private_binding_and_receipt(self):+ cfg,native=self.write_config()+ before=json.loads(native.read_text())["run_bindings"]+ result=h.adopt_handoff(self.evaluate(),cfg,native,self.root/"private-state","fixture")+ self.assertTrue(result["native_stop_hook_bound"])+ self.assertEqual(json.loads(cfg.read_text())["runs"][0]["session_id"],NEW)+ after=json.loads(native.read_text())["run_bindings"]+ self.assertEqual(len(after),len(before)+1)+ self.assertEqual(after[0],before[0])+ self.assertEqual(after[-1]["session_id"],NEW)+ self.assertTrue((self.root/"private-state"/("handoff-"+NEW+".json")).exists())+ self.assertFalse(result["no_adom_submission"] is False)+ def test_adopt_rejects_unverified_handoff(self):+ cfg,native=self.write_config()+ with self.assertRaises(ValueError):h.adopt_handoff({"verified":False},cfg,native,self.root/"private-state","fixture")+ def test_adopt_rejects_mismatched_owner(self):+ cfg,native=self.write_config()+ j=json.loads(cfg.read_text());j["runs"][0]["session_id"]="another";cfg.write_text(json.dumps(j))+ with self.assertRaises(ValueError):h.adopt_handoff(self.evaluate(),cfg,native,self.root/"private-state","fixture")+ def test_adopt_rejects_wrong_ai_thread(self):+ cfg,native=self.write_config()+ j=json.loads(native.read_text());j["run_bindings"][0]["ai_thread"]="wrong";native.write_text(json.dumps(j))+ with self.assertRaises(ValueError):h.adopt_handoff(self.evaluate(),cfg,native,self.root/"private-state","fixture")++if __name__=="__main__":unittest.main(verbosity=2)+
Comments
Log in to comment.