Closed general

Build time: split hydrogen-control (27k-line lib.rs), an insiders profile, and frontend-only builds that skip Rust

John Lauer · 1mo ago ·closed by John Lauer

Plan for the week of 2026-09-07, from John: "record your plan so we can do it next week". Every insiders build today took ~15 minutes, and the numbers say why.

Measured (2026-09-04)

Piece Size
hydrogen-control/src/lib.rs, one file 27,478 lines
hydrogen-app crate 35,079 lines
All Rust, five crates 70,015 lines
Frontend, Svelte + TS 205,182 lines
Dependency crates (Cargo.lock) 643

The build IS incremental at the crate level: the last log shows exactly two crates compiled (the app and hydrogen-control), the other 641 came from cache. The time is the two giant crates in release mode, where intra-crate incremental compilation is off, so a one-line change in the 27k-line file recompiles all of it and then the app crate that depends on it.

The plan, in payoff order

  1. Split hydrogen-control/src/lib.rs into modules, then into crates. Natural seams already exist in the file: the control HTTP server and routing; AI threads (tabs mirror, run-prompt jobs, hooks); permissions (mirror, dismiss, self-heal); wake and workspace health; the code-server/proxy bits. Modules first (mechanical, one PR, zero behaviour change), then crates so a change rebuilds a slice. This is the change that pays back on every build afterwards.
  2. An insiders release profile. [profile.insiders] inheriting release with incremental = true, codegen-units = 16, lto = "thin"; the build task uses it for insiders and keeps the full release profile for public promotions. Expect a large cut on the recompile half of the wall clock.
  3. Frontend-only builds must not touch Rust. Every insiders build bumps version in Cargo.toml, which forces the app crate to recompile even when only Svelte changed. Options: stamp the insiders build number into a file the app reads at runtime (Cargo version stays at the public version), or bump Cargo only when Rust changed. Either way a frontend-only insiders build becomes pnpm + NSIS + sign, a couple of minutes.
  4. Fix the build task itself. C:\tmp\ah-installer-task.ps1 writes exit 0 to its done-file even when tauri build exits 1 (it happened twice today and cost a cycle each time); it appends to one log forever (30k lines, mixed encodings); and it has no timestamps, so the phases cannot be timed. Write the real exit code, rotate the log per build, stamp each phase (pnpm, cargo, bundle, sign) so the next measurement is exact.
  5. Pre-build lints. Ten scripts run before Tauri; measure them once and move any slow one to CI instead of the release path.

What not to do

Do not cargo clean between builds and do not bump the dependency set casually: the 641-crate cache is what keeps this at 15 minutes instead of 45.

12 Replies

John Lauer · 1mo ago

Started today at John's word ("lets do the build-time plan now"). In 1.0.203 (building now):

  • Item 2, incremental release compiles: [profile.release] incremental = true at the workspace root. The other knobs were already at their fast values (Cargo defaults: 16 codegen units, no LTO), so incremental was the missing one. The first build with it is a full one (no cache yet); the measurement is the build after it, against a one-line change in hydrogen-control.
  • Item 4, the build task: scripts/laptop/ah-installer-task.ps1 is now in the repo and installed on the laptop. One log per build under C:\tmp\ah-build-logs\, a timestamp on every phase (frontend, rust+bundle+sign), the frontend failure short-circuits the Rust build, the done-file carries the real exit code, and a "0 but no installer" case is reported as a failure. Phase timings from this build and the next will be posted here.

Still to do: item 1 (split hydrogen-control's 27k-line lib.rs into modules, then crates), item 3 (frontend-only builds that do not bump Cargo.toml), item 5 (time the ten pre-build lints once). Item 1 is being surveyed for seams now.

John Lauer · 1mo ago

Measurements, from the new build task's timestamps (AdomLapper, 2026-09-05):

build change frontend rust + bundle + sign total crates compiled
1.0.203 profile change (cache priming) 204 s 765 s 970 s 9 (tao, tauri, plugins + ours)
1.0.204 ONE comment line in hydrogen-control, incremental warm 141 s 691 s 831 s 2 (hydrogen-control, hydrogen-desktop)

Reading:

  • Incremental release compiles buy ~10% (765 -> 691 s) on a one-line change. Item 2 stays on (it is free) but it does not solve this: the unit of work is still the 27k-line crate plus the 35k-line app crate that depends on it, in release codegen, plus the link, bundle and sign.
  • The lints are not the frontend cost. All eleven pre-build checks take ~22 s here (brand-teal grep 8 s, phantom tokens 5 s, em-dash 4 s, the rest under 2 s). The frontend phase is the Vite build itself (141-204 s). Item 5 closes as "leave the lints alone; if the frontend needs to be faster, it is Vite's bundle, not the guards".
  • So item 1, the crate split, is the fix, and it is the only remaining one that changes the shape of the problem. Step one is done and pushed as branch split-modules: lib.rs 27,478 -> 13,454 lines, six modules (hbw 6.5k incl. an unnamed 1k section, post_dispatch 2.3k, runtime_identity 1.9k, ai_tabs 1.6k, health 1.4k, permissions 0.2k), verbatim moves, cargo check on the crate unchanged. Step two (crates) needs a control-core for the shared statics and helpers every section uses, leaf crates for the sections, and the dispatcher on top; the dependency analysis for that is running now.
  • Item 3 (frontend-only builds skipping Rust): with the numbers above, a frontend-only insiders build currently still pays the app-crate rebuild because the version bump changes Cargo.toml. Worth doing after the crate split, when that rebuild is a small crate.
John Lauer · 1mo ago

The crate split is done and merged to main as 1.0.206 (building now). hydrogen-control is six crates on one core:

  • hydrogen-control-core: the statics, CDP and exec helpers, ports, ad, recorder, safety, the generated manifest table, and the dispatch seam
  • hydrogen-control-health, hydrogen-control-permissions, hydrogen-control-browser (hbw + runtime identity), hydrogen-control-ui (post-dispatch helpers + demo), hydrogen-control-ai-threads (the listener, drafts, tab state)
  • hydrogen-control: the route dispatcher alone (8.4k lines), re-exporting the rest so every hydrogen_control::name path hydrogen-app uses still resolves

The one back-edge (the listener calling the dispatcher) is a fn-pointer seam in core that hydrogen-control installs before it starts the server. No behaviour change; both the Linux check and a local x86_64-pc-windows-gnu check pass, and the laptop builds green.

Measured on AdomLapper, same task, same machine, incremental release profile in all rows:

change frontend Rust + bundle + sign
monolith, one-line change (1.0.204, yesterday's baseline) 141 s 691 s
split, structure change (every control crate recompiles) 89 s 411 s
split, a change in core (worst case: everything sits on core) 85 s 342 s
split, a comment-only change in the dispatcher (the common case) 86 s 333 s

So the Rust step halves, and the interesting fact is that the common case and the worst case are now within 10 s of each other: what is left is a fixed tail (the dispatcher crate itself, the hydrogen-app crate, link, NSIS, signing), not the control code. The 1.0.206 build runs with cargo --timings so the next post has the per-crate breakdown of that tail; the obvious next cuts are the dispatcher's own 8.4k lines and hydrogen-app's setup_steps_wsl.rs.

Item 3 (frontend-only builds) is next after the release: at 85 s the frontend is now a fifth of the total, so it moves down the list.

John Lauer · 1mo ago

1.0.206 is on insiders (installed and running on AdomLapper), and its build ran with cargo --timings. The breakdown of the 346 s Rust step, from target/cargo-timings:

unit time
hydrogen-desktop (the hydrogen-app crate: lib.rs 5.8k, setup_steps_wsl.rs 5.7k, runtime/wsl.rs, wake_heal...) 186 s
hydrogen-control (the dispatcher) 2.0 s
hydrogen-control-browser / core / ai-threads / health / ui / permissions 0.7 to 1.8 s each
cargo total (last unit ends) 196 s
outside cargo: NSIS bundle + Azure Trusted Signing of the exe and the installer about 150 s

So the split did its job completely: the control crates are no longer on the critical path at all. What is left is one crate, hydrogen-app, which recompiles in full every build because build.rs stamps a fresh HD_BUILD_STAMP env into it each time (the env! use invalidates the crate), and then a fixed 150 s of bundle and sign.

Next levers, in order of payoff:

  1. Split hydrogen-app the same way (setup steps, the WSL runtime, wake_heal, the updater into crates on a small app core): the 186 s becomes the size of the crate you touched. This also answers item 3 (frontend-only builds): the frontend is embedded at compile time, so a frontend-only change always recompiles the app crate; making that crate small is the fix, not a separate pipeline.
  2. Move the build stamp out of the env! path (a generated include_str! file that only the version module reads) so an unchanged app crate is a no-op rebuild.
  3. Time the bundle and sign phases separately in the build task and see whether the two Azure signings (~30 s each, serial) can overlap or use a cached token.

Not starting these today; the day's release is out. Left on the branch history for whoever picks it up: the split's mechanics were contiguous moves plus pub(crate)-to-pub widening, with a local cargo check --target x86_64-pc-windows-gnu catching the cfg(windows) paths the Linux check never sees (that found one, setup_live_log, before the laptop did).

John Lauer · 1mo ago

Two more builds today, and a caveat that changes how these numbers should be read: the laptop's ASUS power scheme was on Silent (throttled) for at least the 1.0.207 build and was switched to Performance 74 s into 1.0.208's Rust phase. Compare only within a mode.

build change mode frontend cargo Rust step total (cargo + bundle + sign)
1.0.206 version bump only unknown 86 s 196 s 346 s
1.0.207 dispatcher + frontend Silent 129 s 427 s 628 s
1.0.208 core changed (every control crate recompiles) Performance 103 s (Silent) 270 s 435 s

Per-crate, from cargo --timings (Performance, 1.0.208, the worst case since everything sits on core):

unit 1.0.208
hydrogen-app crate 203 s
hydrogen-control (dispatcher) 43 s
hydrogen-control-ai-threads 26 s
hydrogen-control-browser 20 s
hydrogen-control-core, -ui 11 s each
hydrogen-control-health, -permissions 3 s, 1 s
bundle + sign, outside cargo about 165 s

Readings: the same 11 units ran uniformly 1.5 to 2x slower under Silent (the 1 s crates doubled too), so 1.0.207's 628 s was the power mode, not the code. A dispatcher change costs about 45 s of compile in either build. The app crate at ~200 s and the fixed ~165 s of bundle and sign are now two thirds of the wall clock; those are the next two levers, unchanged from the previous post. The build task could pin the Performance scheme for the build and restore the user's scheme after, if the laptop stays the build box.

John Lauer · 1mo ago

The app split is done and merged (main, going out as 1.0.209): hydrogen-runtime (the WSL runtime, wake repair, editor updates: 6.4k lines) and hydrogen-setup (the cascade, first-run intro, Adom Bridge install/supervision, window probe, plus the thirteen lib.rs items the cascade needed: 10k lines) now sit between hydrogen-control and the app shell, which is down to lib.rs 5.4k + updater + three small modules. Commands moved with their code (a pub #[tauri::command] exports its glue macro; the app glob re-exports the module, so generate_handler! is untouched). Also found and fixed: the build task ran the frontend build TWICE per release, once as its own phase and once again as tauri's beforeBuildCommand inside the Rust phase; the override now skips the second (~90 s per build).

Measured on AdomLapper, Performance mode, cargo --timings:

build app crate unit cargo total Rust step (cargo + bundle + sign)
1.0.208 (control split only, duplicate frontend still on) 203 s 270 s 435 s
+ hydrogen-runtime, duplicate frontend off 160 s 197 s 267 s
+ hydrogen-setup 173 s 222 s 305 s
experiment: same, with the 46.6 MB of .glb held out of the frontend dist 117 s

The surprise is the third row: taking 10k lines out of the app crate did not make it cheaper (hydrogen-setup compiles in 30 s on its own). The app crate's cost is not its code. generate_context! embeds the whole frontend dist, 82.5 MB (46.6 MB of .glb workcell/molecule models, 29.5 MB of JS including the Babylon inspector and hero-deck bundles), and tauri-codegen brotli-compresses every file at quality 9 inside that crate's compile. The hold-out experiment prices it: about 1.2 s per MB, so roughly 100 s of the 173 s is asset embedding and about 70 s is Rust.

So the remaining levers, priced:

  1. Stop embedding the big binary assets (models, maybe the inspector bundle): ship them as on-disk resources (like resources/ab and the skills already are) and serve them to the tauri.localhost origin from ah's local HTTP server, with the shared frontend taking a base URL for them (Web Hydrogen keeps serving them from static/). Saves ~56 s (models only) to ~100 s (all assets). Frontend + control-server change; needs a decision on where the models live.
  2. Turn off tauri's compile-time compression (the compression feature): saves most of the same ~100 s but the exe grows by ~80 MB and NSIS then LZMA-packs it at bundle time, so the net is unmeasured and the installer gets bigger for everyone.
  3. Dead code: 9.4k lines of undeclared files in hydrogen-app/src (screenshot, commands, the bridges, caption) have been commented out of the build since August. Zero compile cost, but they mislead every analysis; delete on a quiet day.
  4. Bundle + sign is now ~70 s after cargo (8 signings at ~4 s, the rest makensis).

Dropped from the plan: the build stamp. A no-change rebuild costs 154 s, but any dependency or frontend change recompiles the app crate regardless, so moving the stamp out of env! only helps the literally-nothing-changed case.

John Lauer · 1mo ago

1.0.209 is on insiders, installed on AdomLapper and smoke-tested (setup state complete through the new probe seam, /workspace/exec on the container route, updater, wake, health all ok). It carries the app split and, per John's call in #60, no 3D model assets in the ah build.

A correction to the previous post first. I priced asset embedding at about 100 s from a hold-out experiment, but that experiment was a no-code-change rebuild (incremental reuse), not a release build, so it overstated the share. With the models actually removed and a normal release build:

build (Performance) app crate unit cargo total Rust step
setup split, models in 173 s 222 s 305 s
1.0.209, models out 161 s 177 s 242 s
no-change rebuild, models in / out (experiment) 154 s / 117 s

So the 46.6 MB of models cost 12 to 37 s of the app crate depending on how much incremental reuse the build gets, not 56 to 100. The bulk of the 161 s is the crate's own release codegen plus tauri's context and command glue, and that is what is left to attack (the remaining 29.5 MB of JS is the next asset lever, most of it the Babylon inspector and XR chunks that leave with the viewer code in #60).

The user-facing effect of the models leaving is larger than the build one: the installer went from 37.5 MB to 19.8 MB, so every insiders and public update download halves.

Where the day ends, one-line change on the laptop, Performance mode:

stage this morning now
frontend 141 s ~90 s
Rust step (cargo + bundle + sign) 691 s 242 s
total 13.9 min 5.5 min

Still open: the Babylon code sweep (#60), the app crate's remaining 161 s, and the 9.4k lines of dead files in hydrogen-app/src.

John Lauer · 1mo ago

1.0.210 (Babylon code gone, dead app files gone) on insiders, Performance mode:

build app crate unit cargo total Rust step
1.0.209 (models out) 161 s 177 s 242 s
1.0.210 (code sweep: dist 21 MB, chunks 9.6 MB) 179 s 226 s 296 s

The app crate did not move (179 s, inside the run-to-run band of 160 to 180 s) even though the embedded assets fell from 36 MB to 21 MB, and cargo total was up because the workspace manifest changed and the control crates re-fingerprinted (core 15 s). Conclusion, now with three data points: asset embedding is a minor share of the app crate; its cost is the release codegen of lib.rs (5.4k lines, 50 commands, the tauri builder and generate_handler!) plus tauri's context. The user-facing wins are real anyway: installer 17.1 MB, and the dead code is gone.

Running now: a measurement build with the app crate at opt-level 1 (per-package override, everything else at 3) to see how much of the 179 s is optimisation. If it is most of it, the choice is between a lower opt level for the shell crate alone (its code is orchestration, not hot loops) and moving the commands below the shell; the numbers decide.

John Lauer · 1mo ago

Build box moved from AdomLapper (John's personal laptop) to ConfRoomROG (2026-09-05 evening), so the laptop's SSD stops eating builds. What the move consisted of, all per-user, no admin on the box:

  • git fetches with John's GitHub credential in Git Credential Manager; the checkout is on main.
  • node 22.21.0 and pnpm 11.1.1 (zip install), dotnet SDK 9 with the sign tool 0.9.1-beta, Az.Accounts.
  • Signing: no Azure CLI on CRR (its installer wants admin), so the task passes -act azure-powershell through the tauri config override and the sign tool uses John's saved Az context (one device-code login). Verified: an unsigned scratch exe came back signed by Adom Industries, Inc.
  • The build task is box-independent now (scripts/laptop/ah-installer-task.ps1, still named for its origin): paths by user profile, per-user toolchain on PATH, pnpm install --frozen-lockfile before the frontend, a High performance Windows power plan created and pinned for the build and the previous plan restored on every exit path, the two-newest-installers prune.

First numbers on CRR (i9-12900H, 40 GB), Windows plan High performance but the ASUS Armoury profile still on Silent:

build frontend Rust step app crate unit
cold (empty target, no Vite cache) 158 s 659 s
warm, no code change 134 s 502 s 312 s

The laptop did the same app crate in 160 to 180 s. The gap is the Armoury Crate Silent profile capping sustained package power; the Windows power plan alone does not lift it, and CRR does not expose the ASUS ATK WMI interface to a non-admin session, so the task cannot switch it. It needs to be set to Performance once in Armoury Crate on the box; then the CRR numbers should land near the laptop's, and the incremental on/off question (dropped on the laptop, which was too memory-starved to measure cleanly) gets measured here.

John Lauer · 1mo ago

Armoury Crate on ConfRoomROG set to Performance (John, by hand: the ASUS profile is not reachable from a non-admin session). Same head, same task, warm cache:

CRR, warm, no code change Armoury Silent Armoury Performance
pnpm install (frozen) 3 s 2 s
frontend 134 s 35 s
Rust step (cargo + bundle + sign) 502 s 152 s
app crate unit 312 s 81 s

So the ASUS profile was worth 3x, not the 1.5x I guessed, and CRR at Performance is now the fastest box we have measured: about 3.2 minutes end to end for a rebuild with nothing changed, against the laptop's best of 5.5. A real one-line change adds the changed crate on top (dispatcher about 45 s, a section 1 to 20 s).

Running now on CRR: the incremental on/off measurement that the laptop was too starved to finish (warm and one-line-change builds with incremental = true, then the same pair with it off). It decides whether release builds keep the 11 GB-a-day incremental cache.

John Lauer · 1mo ago

Incremental verdict, measured on CRR at Performance (cargo build of the app, no bundle), tree restored after:

build cargo
incremental on, warm, no change 122 s
incremental on, one-line change in the dispatcher 85 s
incremental off, first build after the profile flip 179 s
incremental off, one-line change 143 s

Incremental stays: it is worth 58 s (40%) on the change that matters. Its cache was 3.3 GB after four builds and only grows, so the build task now wipes target/release/incremental when it passes 12 GB and lets one build repay it. The task also restores tauri's generated ACL schemas before pulling (cargo rewrites them every build and they were blocking fast-forwards on both boxes).

John Lauer · 1mo ago

Closing: the plan shipped. What landed, all tagged with this issue:

  • hydrogen-control split into six crates in 1.0.206 (core, health, permissions, browser, UI, AI threads); the old 27k-line lib.rs is a 9.9k-line dispatcher.
  • hydrogen-app split into hydrogen-runtime and hydrogen-setup in 1.0.209, plus 9.4k lines of dead files deleted.
  • The build task no longer builds the frontend twice (about 90 s per release) and leaves cargo timings behind for every release.
  • ConfRoomROG on the Performance profile (Rust step 502 s to 152 s) and incremental compilation kept (worth about 40 percent on a one-line change).

A release build on ConfRoomROG is now about five minutes end to end, down from about fifteen.

Log in to reply.