← Commit history

verb suite: section rollup + daily cron + taxonomy in the skill

John Lauer ·295faf32b1 ·1mo ago ·parent 10e622e
3 files changed +155−2
dev-skills/fusion-verb-tests/SKILL.md+41−1
@@ -22,6 +22,41 @@ Exit code 0 = all green. Reports land in `tests/reports/verb-test-<stamp>.json`. Use the ai-thread the runner sets (`fusion-verb-tests`) when investigating by hand, so the user's Activity Log shows the test run as one coherent actor. +## The high-level product sections (John, 2026-08-14 - report against THESE)++A run must be readable as "which product areas are green", not a flat verb list. John's+framing, saved verbatim so it is never forgotten:++> i see the fusion bridge now having several high-level sections:+> - basic driving of fusion gui including all electronics tabs like proj mgr, schematic,+>   2d board, 3d board, library, symbol, footprint+> - fusion cad 3d including bom manager (oliver's feature)+> - mcp server in fusion+> - python add-in in fusion that gives us access to all of fusion's python cmds and+>   electron.run+> - aps fusion feature+> - other fusion api's where we have to be authenticated and does auth still work and did+>   our session tokens stay valid++The runner encodes this as `SECTION_ORDER` / `_SECTION_BY_VERB` (plus two buckets the+list implies but doesn't name: `exports` and `bridge-lifecycle`), stamps a `section` on+every result, and prints a per-section rollup at the end of each run; the report JSON+carries the same rollup under `sections`. When you summarize a run to John, lead with+the section scorecard. A new verb gets a section in `_SECTION_BY_VERB` in the same+commit it gets a plan line.++**Known section gaps (be honest about these when reporting):**+- *gui-driving*: the daily tier proves proj-mgr/schematic/2d/3d tab driving via the+  BQ25792 flow; the LIBRARY, SYMBOL, and FOOTPRINT editor tabs are only exercised+  weekly (open_lbr, capture_library_views, generate_package), and there is no+  dedicated symbol-editor / footprint-editor tab drive yet.+- *mcp-server*: daily only asserts `fusion_mcp_status`; a real MCP round-trip+  (`fusion_mcp_call`) is weekly.+- *auth-tokens*: daily proof is real, not a mock: `fusion_aps_get /project/v1/hubs` is+  an authenticated APS round-trip (token refresh included), and the BQ25792 cloud open+  proves Fusion's own signed-in session is alive.+- *exports*: weekly only (gerbers/BOM/CPL/GLB/stackup write to the scratch dir).+ ## The tiers, and why a verb is where it is  | Tier | What qualifies | Examples |@@ -75,6 +110,11 @@ The runner asserts response shapes. Four things still need eyes each daily run:  ## Cadence -Daily: `--tier daily` (a cron on the container works; see the cron-watchdogs skill).+Daily: **installed** - the dev container's crontab runs `tests/daily_verb_cron.sh` at+11:47 UTC (~6:47am Central). It runs `--tier daily` against AdomLapper, appends to+`tests/reports/daily-cron-history.txt`, and on a red run auto-files an issue on+adom/fusion-bridge with the scorecard tail, so a failure is a filed finding before+anyone asks. The AI judgment ritual (screenshots, timing drift, box-left-as-found)+still happens on the next interactive session; the cron covers the shape checks. Before any release: `--tier weekly` plus a by-hand pass of the manual tier's release-rehearsal subset (make_3d_package, build_library_3d) on a box you can watch.
tests/daily_verb_cron.shadded+36
@@ -0,0 +1,36 @@+#!/bin/bash+# Daily fusion-bridge verb test (see dev-skills/fusion-verb-tests).+# Cron-safe: runs the daily tier against AdomLapper, keeps a dated log, and on a+# red run files ONE issue on adom/fusion-bridge with the scorecard so a failure+# is a filed finding by the time anyone looks, not a log nobody reads.+set -u+REPO=/home/adom/project/adom-desktop-fusion-bridge+LOGDIR="$REPO/tests/reports"+STAMP=$(date +%Y-%m-%d)+LOG="$LOGDIR/daily-cron-$STAMP.log"+cd "$REPO" || exit 1+mkdir -p "$LOGDIR"++python3 tests/run_verb_tests.py --target AdomLapper --tier daily >"$LOG" 2>&1+RC=$?+if [ $RC -eq 0 ]; then+    echo "GREEN $STAMP" >>"$LOGDIR/daily-cron-history.txt"+    exit 0+fi++echo "RED $STAMP (exit $RC)" >>"$LOGDIR/daily-cron-history.txt"+BODY=$(mktemp)+{+    echo "Automated daily verb run went red ($STAMP, exit $RC). Scorecard tail:"+    echo+    echo '```'+    tail -25 "$LOG"+    echo '```'+    echo+    echo "Full log: tests/reports/daily-cron-$STAMP.log on the dev container. Versions at run time are in the readiness line of the log. Triage per dev-skills/fusion-verb-tests: re-run the failing verbs with --only first, a startup race or relay flake re-passes; a repeat failure is the real finding."+} >"$BODY"+adom-wiki issue create adom/fusion-bridge \+    --title "Daily verb run RED $STAMP: $(grep -c FAIL: "$LOG" 2>/dev/null || echo '?') failing verb(s)" \+    --body-file "$BODY" >>"$LOG" 2>&1+rm -f "$BODY"+exit $RC
tests/run_verb_tests.py+78−1
@@ -211,6 +211,69 @@ for v in ("fusion_stop", "fusion_kill", "fusion_signin", "fusion_signin_2fa", "f  PLAN = P +# -- Section map ------------------------------------------------------------+# John's high-level product sections (2026-08-14, verbatim intent): the run must+# be readable as "which PRODUCT AREAS are green", not just a flat verb list.+#   1. gui-driving        basic driving of the Fusion GUI incl. all electronics+#                         tabs: proj mgr, schematic, 2d board, 3d board,+#                         library, symbol, footprint+#   2. cad-3d-bom         Fusion CAD 3D including the BOM manager (Oliver's)+#   3. mcp-server         the MCP server inside Fusion+#   4. python-addin       the python add-in = all Fusion python cmds + Electron.Run+#   5. aps                the APS Fusion feature+#   6. auth-tokens        other authenticated Fusion APIs: does auth still work,+#                         did session tokens stay valid+# Plus two buckets his list implies but doesn't name:+#   7. exports            gerbers/BOM/CPL/GLB/stackup off the board+#   8. bridge-lifecycle   readiness/describe/install/uninstall plumbing+SECTION_ORDER = ["gui-driving", "cad-3d-bom", "mcp-server", "python-addin",+                 "aps", "auth-tokens", "exports", "bridge-lifecycle"]+_SECTION_BY_VERB = {+    "fusion_start": "gui-driving", "fusion_stop": "gui-driving", "fusion_kill": "gui-driving",+    "fusion_get_app_state": "gui-driving", "fusion_dismiss_blocking_dialogs": "gui-driving",+    "fusion_open_cloud_file": "gui-driving", "fusion_open_design": "gui-driving",+    "fusion_show_schematic": "gui-driving", "fusion_show_2d_board": "gui-driving",+    "fusion_show_3d_board": "gui-driving", "fusion_board_info": "gui-driving",+    "fusion_close_window": "gui-driving", "fusion_close_document": "gui-driving",+    "fusion_close_all_documents": "gui-driving", "fusion_get_preferences": "gui-driving",+    "fusion_set_preference": "gui-driving", "fusion_prefs_open": "gui-driving",+    "fusion_prefs_close": "gui-driving", "fusion_open_lbr": "gui-driving",+    "fusion_capture_library_views": "gui-driving", "fusion_generate_package": "gui-driving",+    "fusion_attach_3d_package": "gui-driving", "fusion_make_3d_package": "gui-driving",+    "fusion_build_library_3d": "gui-driving", "fusion_detect_layers": "gui-driving",+    "fusion_new_electronics_from_eagle": "gui-driving", "fusion_update_libraries": "gui-driving",+    "fusion_signin": "gui-driving", "fusion_demo": "gui-driving", "fusion_notify": "gui-driving",+    "focus_invariance": "gui-driving",+    "fusion_inspect_bodies": "cad-3d-bom", "fusion_physical_properties": "cad-3d-bom",+    "fusion_assembly_bom": "cad-3d-bom", "fusion_get_design_info": "cad-3d-bom",+    "fusion_get_parameters": "cad-3d-bom", "fusion_document_info": "cad-3d-bom",+    "fusion_import_step": "cad-3d-bom",+    "fusion_mcp_status": "mcp-server", "fusion_mcp_tools": "mcp-server",+    "fusion_mcp_call": "mcp-server", "fusion_mcp_resources": "mcp-server",+    "fusion_mcp_enable": "mcp-server",+    "fusion_run_modeling_script": "python-addin", "fusion_execute_text_command": "python-addin",+    "fusion_electron_run": "python-addin", "fusion_electron_zoom": "python-addin",+    "fusion_electron_pan": "python-addin", "fusion_electron_select": "python-addin",+    "fusion_aps_status": "auth-tokens", "fusion_aps_get": "auth-tokens",+    "fusion_aps_signin": "auth-tokens", "fusion_aps_set_client_id": "auth-tokens",+    "fusion_export_gerbers": "exports", "fusion_export_bom": "exports",+    "fusion_export_cpl": "exports", "fusion_export_optimized_glb": "exports",+    "fusion_fetch_optimized_glb": "exports", "fusion_board_stackup": "exports",+    "fusion_web_export": "exports",+    "fusion_load_design_rules": "exports", "fusion_set_design_rules": "exports",+    "fusion_readiness": "bridge-lifecycle", "fusion_describe": "bridge-lifecycle",+    "fusion_uninstall": "bridge-lifecycle", "fusion_set_auto_update": "bridge-lifecycle",+}++def section_of(name):+    base = name.split("#")[0]+    if base in _SECTION_BY_VERB:+        return _SECTION_BY_VERB[base]+    if base.startswith("fusion_aps_") or "cloud" in base:+        return "aps"+    return "bridge-lifecycle"++ def main():     ap = argparse.ArgumentParser()     ap.add_argument("--target", required=True)@@ -269,9 +332,23 @@ def main():      passed = sum(1 for r in results if r["status"] == "PASS")     failed = [r for r in results if r["status"] == "FAIL"]+    for r in results:+        r["section"] = section_of(r["verb"])     stamp = datetime.datetime.now().strftime("%Y-%m-%d_%H%M")+    rollup = {}+    for r in results:+        s = rollup.setdefault(r["section"], {"PASS": 0, "FAIL": 0, "MANUAL": 0, "SKIP": 0})+        s[r["status"]] += 1+    print("\nSection coverage (John's product map, 2026-08-14):")+    for sec in SECTION_ORDER:+        s = rollup.get(sec)+        if not s:+            continue+        badge = "RED" if s["FAIL"] else ("green" if s["PASS"] else "untested-this-tier")+        print("  %-38s %d pass / %d fail (%d manual, %d skip)  %s"+              % (sec, s["PASS"], s["FAIL"], s["MANUAL"], s["SKIP"], badge))     report = {"date": stamp, "target": a.target, "tier": a.tier,-              "pass": passed, "fail": len(failed), "results": results,+              "pass": passed, "fail": len(failed), "sections": rollup, "results": results,               "durationSec": round(time.time() - t0, 1)}     out = REPO / "tests" / "reports" / f"verb-test-{stamp}.json"     out.write_text(json.dumps(report, indent=1))