Open general

Proposal: crash flight recorder — per-process memory history ring buffer, crash bundles on abnormal events, UI breadcrumbs

Drew Owens · 19d ago

Type: Feature request (proposal)

Context: Two recent investigations hit the same wall.

  • The 2026-07-20 OOM: a claude process died at 15.2 GB. The kernel log names the victim and its final size, but nothing recorded how fast it grew, over what period, or what it was doing. The trigger is unrecoverable.
  • The 2026-06-30 lock-up (#51): heartbeats proved the process was alive, but there was no record of UI-thread health, and the Codex-sidebar trigger appeared in no log at all.

Each time, the post-mortem answer was "the data was never captured." Proposal: a flight recorder for the HD stack, cheap enough to run always, so the next crash comes with its own history.

1. Per-process memory history (the gap this week). A ring buffer, sampled every 10-15 s inside the distro: timestamp, total used, and the top-N processes as (pid, name, RSS). Plain JSONL, rotated at a few MB. Fifteen seconds of granularity over the last hour answers the questions we could not: when growth started, linear leak vs. sudden allocation, what else was under pressure. The memory-watchdog proposal (filed alongside) produces these samples anyway; this just persists them.

2. Crash-context capture on every abnormal event. When HD sees a workbench reload it did not initiate, an AD respawn, a teardown panic, or a child-process exit, snapshot into a crash bundle: last N lines of the ring buffer, kernel OOM block if present, process tree, HD build stamp, and uptime. One directory per event, e.g. %APPDATA%\hydrogen-desktop\crashes\<timestamp>-<kind>\.

3. Structured breadcrumbs for user-visible actions. The Codex lock-up was undiagnosable partly because sidebar/panel opens are not logged. A one-line breadcrumb per UI action (panel open/close, tab switch, bridge call) in the existing log gives crashes a preceding-action trail. Near-zero cost; string volume comparable to the current [focus] lines.

4. Retention + surfacing. Keep the last 10 crash bundles. List them in the About dialog or a hd doctor command, so "grab logs" becomes "grab the bundle for yesterday 12:59" instead of an AI spelunking dmesg and a 1.6 MB log by hand (the current workflow, which only works when someone thinks to ask).

Scope note: the ring buffer + OOM capture live in the distro (image or hd-workspace-tooling); the bundle assembly and breadcrumbs live in HD; AD could adopt the same bundle format for its own respawns. Happy to split this into per-component issues if that's easier to schedule.

1 Reply

John Lauer · 7d ago

Status (2026-08-03): still wanted, not yet built. Two of its motivating strands advanced separately: HD now logs sample-prompt routing decisions, and the tao teardown panic got its own issue (#27) with a fresh 2026-08-03 capture. The ring-buffer + crash-bundle design remains the right shape; queued with #14 for the image/tooling roadmap. Keeping open.

Log in to reply.