Hardware Test and Triage

@@ -0,0 +1,234 @@+---+name: hardware-test-and-triage+user-invocable: true+description: >-+  Bench methodology for hardware test campaigns: writing and hardening test+  procedures against live bench results, triaging failures to root cause+  (hardware vs firmware vs config vs observation error), reading failure+  signatures, designing discriminating tests, and the documentation ledger+  that keeps a multi-day campaign coherent. Distilled from the Control Panel+  Workcell V2.0.1 campaign. Use when running or writing a hardware test+  procedure, bringing up a board, triaging a bench failure, root-causing an+  erratum, or reviewing a test document. Trigger words: "bench test", "test+  procedure", "board bring-up", "triage this failure", "root cause", "why is+  this failing", "hardware test", "errata", "test campaign", "review this+  section", "bench this".+---++# Hardware Test and Triage++Methods proven across a multi-day CM5-carrier + RP2350-scaffold test+campaign that root-caused two design errata, one silicon corner case, and a+dozen procedure defects. The theme throughout: **evidence beats recall,+discriminating tests beat theories, and the observation channel is a+suspect until proven.**++## 1. The document ledger++A campaign needs three documents with a strict promotion gate:++- **The frozen procedure** — never edited mid-run (operator's handwritten+  notes must stay aligned to it). One exception class: pure metadata.+- **The editor (staging)** — ALL draft changes land here, always. "Update+  section X" means update the editor. It doubles as the rolling bench doc;+  prune completed material when moving on.+- **The final-changes ledger** — merge-ready numbered Changes, promoted+  ONLY on the explicit word "send". Each Change carries the corrected text+  plus a *Why:* footer: rationale per deviation AND the bench records that+  prove it (dates, measured values, PASS/FAIL). Amendments to earlier+  Changes are new Changes with an "Amends Change N" header — never silent+  edits.++At run conclusion: merge all Changes into the procedure, regenerate the+PDF. The ledger IS the review artifact; a reader should never need the+conversation.++## 2. Verify before trusting++- **Netlist vs datasheet, every control pin.** The campaign's worst+  erratum: a symbol drawn from the SSOP pin column placed on a VQFN+  footprint — all 28 pins misplaced, chip unpowered, forward-looking+  reviews all passed because net names *follow the symbol's belief*. Check+  function→physical-pin against the datasheet's package-specific column,+  and note whether the part has multiple pinout versions per package code.+- **Cross-reference a proven design.** A working reference board using the+  same part (found on the wiki/registry) settles a pinout dispute faster+  than argument — diff its pad→net map against yours.+- **Datasheet claims have edges.** "True reverse blocking" held only from+  the never-forward-conducted state; once enhanced, output back-bias kept+  the FETs latched on. When bench contradicts datasheet, characterize the+  boundary conditions — both can be "right" in different states.+- **Search, don't recall** for platform behavior (kernel quirks, driver+  bugs, package availability). Known field issues (e.g. iwlwifi INIT -110+  on Pi platforms) save days.++## 3. Failure-signature reading++| Signature | Meaning |+|---|---|+| Register reads all `0x5a5a5a5a` / `0xffffffff` | Device unpowered, unclocked, or held in reset — bus alive, core absent |+| SPI `osc=0xffffffff`, MISO stuck high | Chip not responding at all — power/clock/pinout before protocol |+| Driver `-110` timeout retry storms at boot | Device present but can't complete init — and the retries themselves can stall boot (udev-ordered) and inflate current draw |+| `cannot read IDR` on SWD | Target in hardware reset — check strap/enable GPIOs before blaming the probe |+| Voltage = supply − 0.3…0.7 V | A body diode is conducting; count the junctions (but drops compress at µA DMM loads — don't count from high-impedance readings) |+| Statically open (MΩ) but conducts amps live | An ACTIVE mechanism turns on under bias — not a short; find what powers/enables it |+| Crest factor ~4+ | Noise-dominated capture; a sine reads ~1.4–2.5. RMS Pk ≈ RMS Tr = steady content |+| Full-scale spikes only at capture start | Settle transient (e.g. mic decimation filter) — trim it, don't chase it |+| "Impossible" throughput/speed | The measurement is wrong: cache not flushed, wrong output field read (`Milliseconds` is a remainder, not a total), or wire-rate physics violated |++## 4. Discriminating tests++Design the ONE measurement that splits the hypothesis space:++- **Load test** separates phantom voltage from real conduction: a 10 kΩ+  (then ampere-scale) load collapses a leakage-fed node instantly; a real+  path holds.+- **Fresh-state vs post-conduction** separates architectural blocking from+  state-dependent latch-up: test both orders.+- **Point-blank stimulus** (finger snap at a mic, known tone) separates a+  dead transducer path from a weak source in one 5-second capture.+- **Sequence-swapped runs** (rail off at boot vs on; driver blacklisted vs+  auto) separate ordering races from static faults — but only change ONE+  variable per boot.+- **Timeline correlation** for "when did this break": map every boot/run+  to good/bad, align against the change log. Beware: a "transition event"+  can be an illusion — the fault may have existed from day one, observed+  only when someone finally read the journal.++## 5. The observation channel is a suspect++More campaign hours were lost to broken observation than broken hardware:++- **Ping false-fails**: Windows drops inbound ICMP by default. Prove paths+  with a short TCP transfer, never ping alone.+- **Enumeration latency**: `lsusb`/`lsblk` immediately after a mux switch+  or plug event reads as failure. Sleep 3–5 s; write it into procedures.+- **Buffered tails**: an unopened tty buffers output; the first read+  drains history that looks live. A mid-line truncation timestamps the+  reset that killed the writer.+- **DMM impedance**: 10 MΩ inputs float dead rails up through sense+  networks. Sub-µA "voltages" are not power.+- **The channel and the fault can share a cause**: when SSH is your+  observation channel and networking is the variable under test, every+  conclusion is confounded. Move to UART/serial before concluding anything.+- **Scope bandwidth is part of the measurement**: BW limit OFF for digital+  edge quality (the limit hides the overshoot/runts being judged); 20 MHz+  ON for power-rail ripple (the industry convention — full BW over-reads).+  Ground spring, not the clip lead (the loop shows fake ringing).++## 6. Bench safety rules++- **Never optimize a working system mid-campaign.** The quirky-but-working+  state IS the baseline. Document the quirk in the procedure; fix it in+  the next hardware/image revision. Any system-state change needs a stated+  rollback plan and the owner's explicit go-ahead first.+- **Every ON gets an OFF script, in the same step.** Never assume the+  operator knows how to reverse a state.+- **GPIO/strap state does not survive power events.** Any step that+  depends on enables/straps re-asserts them (idempotent), and the failure+  signature of forgetting is documented next to the command.+- **Define end-state.** A section that leaves a mux switched, a rail on,+  or firmware running says so — and restores or hands off explicitly. If+  the next section needs a blank MCU, the previous section's last step+  erases it.+- **Respect hard limits in-line** (⚠ voltage ceilings that have destroyed+  parts get repeated at every step that approaches them).+- **Destructive-class tests live in an appendix**, sacrificial hardware+  only, never in the per-unit flow.+- **Clean shutdown only** (`poweroff` + PSU off); never hard-kill a live+  filesystem. Back up the configured test image once it carries real+  accumulated state.++## 7. Writing procedure steps++- **Every step self-contained**: instruction + inline script. No "run the+  step from section X" cross-references mid-flow.+- **Pass/fail readable from output**: name the exact lines/values that+  mean PASS and the known failure signatures with their causes. An+  operator should never need judgment where a string match will do.+- **Placeholders carry substitution rules** (`sda1 — substitute from+  lsblk`; `<board-ip>` — bare address, no /24) and a worked example with+  real values sits under the placeholder form.+- **Concurrency is explicit**: mark [BLOCKING ~N min]; say when a second+  terminal is needed and when it is not; and when two things must overlap,+  make them ONE paste (background `&`, settle sleep, `wait`) so sequencing+  mistakes are impossible. Clear the log (`dmesg -C`) at the window's+  start so anything found afterward is guilty by definition.+- **Partner-side commands match the partner's OS.** A Windows link partner+  gets PowerShell (`Get-Volume`, `Measure-Command`, `fsutil`), not `lsusb`+  and `dd`.+- **Embed bench-reference values** ("bench reference: −44 → −47 dBm,+  3 dB") so the next operator can calibrate expectations.+- **No troubleshooting trees in the procedure.** If a failure is a design+  issue, it gets fixed in hardware and the procedure assumes working+  hardware. Diagnostics belong to engineering notes, not the per-unit flow.+- **Gates match physics**: calibrate thresholds to what the test needs and+  the bench achieves (margins, realistic drops), not round numbers; and+  never gate on something a variant/idle state legitimately changes+  without saying so.++## 8. Test firmware patterns (MCU-on-DUT)++- Distinguish **simultaneous-toggle stress** (worst-case SSO load for rail+  ripple) from **walking-1** (per-contact identification) — different+  tools; never share a name.+- **Heartbeat** (LED + UART counter) = liveness with zero ambiguity.+- **PIO/bit-level mirror** for UART echo = baud-agnostic; one image covers+  the whole rate sweep and tests the physical path, not a peripheral+  config.+- Kit discipline: every image ships as elf + uf2 + source, staged at a+  known path, with an `ls` presence check before any step depends on it.+  Transfers to other machines get byte-size verification.+- Firmware persists in flash; enables don't. Say which is which wherever+  it matters.++## 9. Root-cause honesty++- Partition every failure: hardware / firmware / config / observation.+  Change hypotheses when the evidence does — and when the owner asserts a+  constraint ("it is NOT hardware"), re-derive under it seriously; they+  usually know something.+- A revert to known-good is a full revert — inventory every artifact your+  own instructions created (config lines, blacklists, dropped files) and+  audit files you "assumed stock" by reading them, not naming them.+- Own wrong calls explicitly and record what the failed fix taught; a+  disproven hypothesis with data is progress, a defended one is debt.+- The best close is the mechanism that explains EVERY observation at once+  — keep digging until the contradictions are gone, then write it down+  where the next person will find it.++## 10. Run conclusion: the evidence merge++The campaign ends with two documents, not one:++- **The procedure, next Rev** — all ledger Changes merged, renumbered, with+  the run's measured values embedded as "bench reference" calibration+  values at the step that records them. Sections that could not execute+  (blocked by errata/missing hardware) merge as TENTATIVE with the blocker+  and the planned fix stated in the section header block.+- **A separate evidence appendix** — every scope capture and script+  screenshot paired to the step it evidences, captioned with the value the+  procedure records. Dangerous/engineering-only steps stay in the+  PROCEDURE's appendix; the evidence document is images + captions only.++Mechanics that made it clean:++- **Title captures with the step number at capture time.** When the merge+  renumbers sections, publish an as-shot → final numbering map in the+  appendix and quote each original filename verbatim — traceability+  survives the renumber.+- **Captures are records, not illustrations.** Where the run notes lack a+  number, read it off the capture's on-screen measurements and back-fill+  the bench reference; where a filename suffix is ambiguous, show the+  filename and describe only what is on screen — never interpret.+- **OCR of handwritten notes is a suspect** (§5 applies to paper too):+  cross-reference the converted text against the source scan before+  trusting any value it carries.+- **Expected noise gets captioned.** If a passing run's screenshot is full+  of scary-looking but documented-cosmetic output (stderr spam, retry+  lines), the caption says so and points at the procedure text that+  documents it — otherwise the evidence contradicts the record.+- Audit the merge mechanically: every internal cross-reference re-checked+  against the new numbering, every image referenced exactly once, leftover+  ledger headers grepped out, placeholders remaining only in sections that+  legitimately never ran.