Open general

Startup network burst multiplied by restarts: about 2 requests/s per crash-looping install (4x carbon /user per launch, installer re-download, no saved backoff)

John Lauer · 1mo ago

Colby caught Hydrogen Desktop hitting carbon.adom.inc/user about 4 times a second per installing machine (fixed in ah 1.0.571). We then audited every Adom client for the same class of bug. Bridge is quiet in steady state (about 15 to 30 requests an hour per install). The danger is startup cost multiplied by restarts. One launch from Hydrogen makes about 25 to 30 remote requests, and nothing about that rate is remembered on disk. Hydrogen respawns a crash-looping Bridge about every 15 s (about 240 launches an hour), so one crash-looping install sends roughly 6,000 to 7,500 requests an hour (about 2 a second):

Destination Per launch Per hour in a crash loop
carbon.adom.inc 4 x GET /user + 1 PATCH about 1,200
wiki.adom.inc about 15 to 20 about 3,600 to 4,800
fleet pings (insiders) 5 about 1,200
adom.cloud 1 relay WS upgrade per cloud server, plus the mesh relay about 240 each

If a Bridge update is pending, every launch also re-downloads the whole ~22 MB installer, up to about 5 GB an hour per install.

Audited origin/main at c0fde4ed (2.1.170), code reading only. Paths are under src-tauri/src/ unless they start with src/main.js. (Hydrogen's side of this, the 15 s respawn with no backoff, is ours to fix in ah; we'll do it there.)

1. Every Hydrogen spawn runs an update check at once, and can re-download the installer

  • Trigger: Hydrogen passes --session-token on every spawn, so auth.rs:202 sees a handoff. lib.rs:5903 calls accept_handoff (auth.rs:265), which calls updater::spawn_check_now("handoff") (auth.rs:286), skipping the updater's 20 s delay.
  • Requests: the wiki version.json (insiders plus public, updater.rs:567, :600). With an update pending: a HEAD and Range GET (updater.rs:723), then the full installer (updater.rs:532, download_and_stage). For insiders: fleet POST /ping for Bridge and for each bridge (updater.rs:832, :919). With the bridge cache always empty at launch, each bridge's manifest is re-checked too (updater.rs:928).
  • Why it multiplies: "already staged" lives only in memory (updater.rs:1855, status created fresh at lib.rs:6113). spawn_check_now ignores update_mode=off and embedded mode. POST /embedded/enter with a token (direct_api.rs:2985-3015) runs spawn_check_now and kick_check_after_session back to back, and only kick_check has an in-flight guard.
  • Fix: save the last check time and the staged installer's version and hash to disk. Skip the handoff check inside about 30 minutes of the last one, never re-download an installer already staged with a matching hash, and make spawn_check_now respect update_mode=off.

2. Carbon /user is read 4 times per launch, plus a PATCH

  • accept_handoff (carbon.rs:157), cloud server pull (lib.rs:5926 to cloud_config.rs:91), mesh relay token (mesh.rs:363 to :1962) and the mesh peer directory (mesh.rs:481 to :534) each GET /user for the same document. Then PATCH /user/hydrogen/settings (carbon_runtime.rs:156, started at lib.rs:5722).
  • Fix: fetch /user once per launch and share it. Skip the PATCH when the record is unchanged and was published under 4 h ago (keep a copy on disk).

3. The UI re-checks every bridge on the wiki 4 s after each page load

  • src/main.js:4511-4515 calls check_bridge_updates (lib.rs:2620 to bridge_cache.rs:1699) 4 s after load, then hourly. The page loads on every launch (including hidden and embedded ones), after each ~10 minute memory-saving unload, and on webview crash recovery. This skips the background sweep's jitter. About 2 GETs per bridge per load.
  • Fix: read the cached status on load, and keep the live check for the "Check for updates" button.

4. Relay WebSocket reconnect: no jitter, and fast reconnects after longer connections

  • Backoff is 1 s to 30 s for quick failures and 5 minutes on auth failures (ws_client.rs:232-325), which is good. But a clean close after more than 10 s reconnects immediately (:276-281), and an error after more than 10 s waits 1 s (:300-305). A relay or proxy that drops connections at 10 to 15 s therefore gets a reconnect every ~11 s forever (300 to 360 an hour per server). There's no jitter, so after a relay kick_all every Bridge reconnects in the same instant.
  • Fix: add ±20 to 50% jitter, treat connections under about 60 s as short, and keep escalating across repeated short connections.

5. Relay peer list polled every 12 s while the window is showing

  • src/main.js:2390 calls relay_peers (lib.rs:1811), which POSTs get_clients to the relay's 8766 through the adom.cloud proxy, once per relay every 12 s (about 300 an hour each).
  • Fix: poll every 60 s or more, or push peer changes over the existing WebSocket.

Lower

  • 6. web_demo.rs:185 to :67 fetches web-control-demo.html from the wiki on every launch.
  • 7. POST /update/check (direct_api.rs:3248, verb at commands.rs:2903) has no throttle. Each call runs a full check plus fleet pings.
  • 8. The runtime record publish retries every 60 s forever on a 429 or 5xx (carbon_runtime.rs:149-168), so the whole fleet hits Carbon in sync during an outage. It needs exponential backoff with jitter.
  • 9. Per-verb bridge auto-update (bridge_autoupdate.rs:23,45): the 30-minute throttle is in memory, so every new process repeats it.
  • 10. Bridge hero image and wiki metadata (lib.rs:2932-3060, lib.rs:3270+) have no negative cache. A bridge with no hero is re-fetched (2 to 6 GETs) on every page load.
  • 11. After a failure, the runtime download (runtime_cache.rs:283, :569-574), missing-bridge reconcile (bridge_cache.rs:493) and first-run prewarm repeat on every launch with no saved backoff.
  • 12. Hourly cloud sync (lib.rs:7007 to :4150-4160) sends an unconditional PATCH plus GET /user, even when nothing changed.

Suggested crash guard

Record launches on disk. If Bridge has started more than N times in M minutes, skip all optional startup network work (demo page, fleet, bridge checks, reconcile, prewarm) until it stays up.

Already fine

  • The background bridge sweep (jittered, 20 to 28 h).
  • The 4 h updater.
  • Daily skill sync.
  • Mesh peer reads (throttled to one per 20 s).
  • Sign-in polling (2 s, capped at 14.5 minutes).
  • The container-side relay supervisor (backs off).
  • One open question: iroh's mesh relay reconnect backoff on a 401 could not be verified locally.

1 Reply

John Lauer · 1mo ago

Thanks for the audit. 2.1.171 (insiders now) takes the items that multiply with restarts:

  1. Hand-off update check: throttled to once per 30 min by a timestamp saved on disk, which survives restarts. It also respects update_mode=off; the 4-hourly supervisor still reads the manifest for the emergency signal. A real sign-in is not throttled. An installer already in %TEMP% whose sha256 matches the manifest is adopted as staged, with no re-download.
  2. Carbon /user: one shared read. Every caller in a 10 s window gets the same response, and concurrent callers wait for it instead of each fetching. It is keyed by the token, and a settings PATCH drops the cached copy so the next read sees the write.
  3. UI bridge check on page load: answered from a result saved on disk within the last hour. The Refresh button forces a live check, and an applied bridge update clears the saved result.
  4. Relay reconnect: ±25% jitter. A connection now counts as long-lived only after 60 s (was 10 s). A clean close after a long connection reconnects in about 2-4 s, jittered, not in the same instant as every other box.
  5. Relay peer poll: 60 s (was 12 s).
  6. Runtime-record publish: exponential back-off with jitter (60 s up to 1 h), instead of every 60 s forever.
  7. Crash guard: launches are recorded on disk. Five launches within five minutes skips the optional startup network work (ecosystem skill fetch, demo-page refresh, missing-bridge reconcile, bridge prewarm) until ab stays up. Connections, relay, mesh and the update supervisor are unaffected.

Still open:

  • #7 throttle on /update/check.
  • #9 persisted per-verb auto-update throttle.
  • #10 negative cache for heroes.
  • #11 saved back-off for the runtime download, reconcile and prewarm (the crash guard covers the storm case).
  • #12 the unconditional hourly cloud sync.

Leaving the issue open for those.

Log in to reply.