Closed bug report

send_files over the mesh: a multi-chunk send exits 1 with no output after ~60 s while chunks crawl at ~6 s each (was 0.4 s)

John Lauer · 1mo ago ·closed by John Lauer

Seen 2026-09-20 21:30Z while staging a build from AdomLapper to arav-rog with the Windows CLI (2.1.138 build, same send code as 2.1.137) through the local ab's mesh egress (send_files --target arav-rog, the 2.1.97 write_file_chunk route).

Symptom. A one-chunk file (11 bytes) works. Any multi-chunk file makes the CLI exit 1 with NO output at all (stdout 0 bytes, stderr 0 bytes, RUST_LOG=debug adds nothing), from Git Bash and from the Windows shell alike: a 5 MB file (7 chunks) after 27-56 s, a 49 MB file after 61 s. arav-rog's log shows the chunks ARRIVING, one every ~6 s (768 KiB per chunk, about 1 Mbit/s), then the stream simply stops. Earlier the same day the same route moved 63 chunks in 15-35 s (0.3-0.5 s per chunk), so two things changed at once: the per-chunk latency went up ~15x and whatever budget the CLI applies to the whole send is hit and reported by a silent exit.

Asks.

  1. The CLI must never exit non-zero without a JSON error on stdout. Whatever path ends the send (a wall-clock budget, a watchdog, a chunk timeout) has to print {success:false, errorCode, error, sentChunks, ofChunks, lastOffset} so the caller learns where it stopped.
  2. The send's budget must scale with the file (chunks x per-chunk allowance), not be a fixed number, and the reply should carry the measured per-chunk time so a slow mesh path is visible.
  3. Find out why the mesh chunk rate dropped from ~0.4 s to ~6 s per chunk between 16:00Z and 21:30Z on the same two boxes (both on 2.1.137 by then; the laptop had auto-updated in between): desktop_mesh_status on both, the path (direct vs via Adom relay), and whether the relay-carried path throttles.

Workaround used: deploy through the signed installer via update_check / apply_update instead of staging the exe over the mesh.

1 Reply

John Lauer · 1mo ago

Root cause found, and it is not in ab.

The silent exit 1 was Hydrogen's ad_supervisor::check_ad_version_drift ("STALE AB RECLAIM"): on every health-poll tick it force-kills any running adom-bridge.exe whose path is not the installed GUI exe. The CLI's cargo bin is also named adom-bridge, so a CLI run from the build tree on a Hydrogen box is killed within ~60 s of starting, mid-call, with no output. Hydrogen's own log on AdomLapper has one RECLAIM line per dead send (16:29, 16:30, 16:31, 16:32, 16:34, 16:35, 17:11, 17:15 local), and a byte-identical copy of the CLI renamed ab-test-cli.exe ran a 70 s call to completion. Filed as adom/hydrogen#88 with the evidence and three fix options (kill only a real port holder; skip console-subsystem exes; match the GUI by version-resource ProductName).

What that means for the asks here:

  1. The CLI had nothing to print: it was terminated, not failed. Kept anyway from this investigation: ADOM_SEND_TRACE=1 narrates every chunk (offset, size, answer time) on stderr, so a send that dies mid-stream says where (2.1.139).
  2. No budget was hit; the send loop has none beyond the 180 s per-chunk call.
  3. The chunk rate did not regress in ab: the mesh path between the two boxes was relay (AdomLapper runs relay mode) and the relay's first-byte latency measured 330-1400 ms at the time (connect 90 ms), i.e. ~500 ms RTT per hop, several hops per 768 KiB chunk. Environmental; the relay container was slow this evening.

Installed users are not affected (the installer ships the CLI as adom-bridge-cli.exe, which the reclaim never matches). Closing here; the fix belongs to Hydrogen.

John Lauer closed this · 1mo ago

Log in to reply.