Closed general

Setup telemetry v2: find out why installs fail and where people give up

John Lauer · 15d ago ·closed by John Lauer

Why

Setup telemetry v1 (1.0.512+) tells us which step failed. It does not tell us why, and it cannot tell a slow download from someone giving up, a crash from a quit, or an installer that never got as far as opening Hydrogen. Our most common failure is ensure_workspace_failed_import right after a fresh WSL install, and today we cannot say whether those machines have virtualization off, the VM platform not ready yet, or something else.

The fleet dashboard already accepts and renders every signal below (deployed 2026-10-07), so each one shows up the moment a build sends it. Ship them in any order; the list is ranked by how much each one tells us.

Wire rules (same as v1)

  • Same transport: POST to the fleet_url from the manifest, same body as today (product, version, tier, machine, os, install_id, user_id/username once signed in) plus event and the fields below.
  • All values are strings, each at most 120 characters. Fire and forget, never block setup on a ping, never retry more than once.
  • install_id on every event, so all of it joins into one attempt.
  • None of these events count as an update check (the dashboard already treats anything with event set as funnel only).

1. Sub-phases inside ensure-workspace (event: setup_phase)

ensure-workspace is 3 different jobs. Ping each one with step: "ensure-workspace", phase, state: start | ok | fail, ms on ok and fail, error on fail.

phase where (hydrogen-runtime wsl.rs fresh_import)
wsl_install wsl --install and its UAC prompt
image_download download_tarball
image_checksum verify_tarball_sha256
wsl_import wsl --import Adom-Workspace ... --version 2
first_boot first start of the distro until code-server answers

On fail, also send exit_code (the process exit code) and hresult (the Wsl/Service/... or 0x8037... code from stderr, if present). That one field should answer the import question: for example 0x80370102 usually means virtualization is off in firmware.

2. Download progress (event: progress)

During image_download, every 10% (not more often): phase: "image_download", pct, bytes, total, bps (bytes per second averaged over the last 10%). The dashboard turns this into Mbps and shows finish rate by connection speed, and the exact percent a person was at if they quit.

3. Heartbeat (event: heartbeat)

While any step is running, every 60 seconds: step, phase and pct if known. Without it, a slow 20-minute import and an abandoned window look identical. The dashboard calls an attempt "in progress" while heartbeats are under 3 minutes old and "went silent" after that.

4. Quit during setup (event: quit)

When the window closes or the app exits before setup_complete: step, phase, pct. Send it synchronously with a short timeout (500 ms) before exit. This is the difference between "they gave up" and "it broke".

5. Crash report on next launch (event: crash_report)

Write the current step/phase to a small marker file when each starts, delete it on setup_complete and on clean quit. If the marker exists at the next launch, the previous run died: send crash_report with prev_step, prev_phase, then continue as normal.

6. System profile on first launch

Add these fields to the existing first_launch ping (and only that one):

field value
win_edition Home, Pro, Enterprise, Education, ...
win_build e.g. 26100
ram_gb total RAM, whole GB
disk_free_gb free GB on the drive the workspace goes to
virt_fw yes / no, firmware virtualization (the existing pre-check in setup_steps.rs 678-688 already computes this)
hypervisor yes / no, hypervisorPresent
wsl_present yes / no, before we do anything
wsl_version from wsl --version if present

The dashboard shows finish rate by each of these.

7. Installer and uninstaller pings (event: installer, event: uninstall)

Today we only hear from people once Hydrogen opens. The NSIS installer should:

  • create the install_id (GUID) and write it to %APPDATA%\hydrogen-desktop\install-id.txt; the app reuses it instead of making its own when the file exists,
  • ping installer with state: start when it begins and state: ok or state: fail (with reason, e.g. disk_full, cancelled, access_denied) when it ends.

The uninstaller pings uninstall with the same install_id (and reason if we ever add a "why are you leaving" box). The dashboard then shows "installed, never opened" (installer ok, no first launch after 30 minutes) and uninstalls.

8. Slow step (event: slow)

When a step or phase passes 3x its expected time, ping once: step, phase, ms so far, expected_ms. Suggested expectations: image_download 300000, wsl_import 90000, first_boot 60000, everything else 30000.

9. Error screens and what people press (event: error_screen)

When setup shows an error to the person: screen (a stable id like virtualization_off, import_failed, no_network), step. When they press something on it, send again with button (retry, help, quit). This tells us whether our recovery screens actually recover anyone.

10. Sign-in screen and first use

  • event: signin_screen with state: shown | opened_browser | cancelled | timed_out, reason if any. Today the sign-in wait is only a span; this says whether they bounced off it.
  • event: first_use with action the first time each happens after setup: opened_editor, first_ai_prompt, opened_project. Once each, ever, per install.

11. Exact sign-up join (needs the auth side)

Pass install_id as a query parameter on the sign-in URL Hydrogen opens, so the account created from it records which install it came from. Then a new sign-up joins to its install exactly, instead of us matching by time and IP. This needs a small change on the auth side too; just add the parameter on this side and note it here.

Real attempts to check against once it ships

Import failures right after a fresh WSL install (the hresult from item 1 should explain these):

  • 559561a2-b04f-4911-a9a6-60dda65f255c
  • e8beae4d-262b-4d28-bd33-ec595360494f
  • 20283108-0fd3-4561-961d-9f75fe199480
  • e459a7de-4344-4d4c-bea6-75da2489af18
  • 86472acc-38aa-4653-8506-b88b8463f03e (our test VM with virtualization deliberately off: expected virt_fw: no)
  • 76b8b50e-44fd-4ad9-8eec-c974b4111412 (wsl_install_exit_1)

Went silent at the start of ensure-workspace with no failure (heartbeat, quit and crash_report from items 3 to 5 should explain these):

  • a8cf1aeb-1bbf-4057-af0b-eb174416b1a4
  • 66dbc780-7caf-44f0-88f6-a877c09f5513
  • f42669d9-3ca9-47ad-bdb7-97754dbd0d25
  • d53e14c2-1358-49b5-a1b0-9a9c8624fac6

Done when

Ship to insiders, run one install on a clean Windows VM, and check the Setup funnel tab shows the system profile, the download speed, the sub-phase it stopped at, and the new sections. Reply here with the build number and the install_id of that test and I will confirm each signal landed.

10 Replies

John Lauer · 15d ago

Shipped to insiders as 1.0.570, all 11 items (commit 0914fd6c).

Clean install: install_id 9b88f11a-0e45-4e2e-b78a-9f217501fa98

  • Machine: Azure VM winvm, reset to a clean Windows 11 snapshot with no WSL and no Adom software. Installed interactively from the published 1.0.570 installer.
  • Path: WSL2 installed (wsl_install phase), one restart, workspace built.
  • Where it stopped: the Adom sign-in. The snapshot's adom.inc session has expired, so it is sitting on the login page. That gives the "stopped at" view a real case: step activate-editor, a wait span of kind signin, and heartbeats still arriving.
  • What the app sent from this install, all HTTP 200 from the fleet, 57 pings in total:
    • first_launch with the system profile
    • setup_step
    • setup_phase: 6 phases, each with start and ok: image_download, image_checksum, wsl_install, wsl_import, first_boot, plus the post-restart re-check
    • progress: 10, one per 10%. Azure pulled the 813 MB image in about 6 s, so bps is high.
    • heartbeat every 60 s
    • slow: once, for wsl_install past 3x 30 s
    • wait: uac, reboot, logon, signin
    • signin_screen shown and opened_browser
  • Installer pings: the installer created install-id.txt, and the app reused that id. The installer's own installer start/ok pings are fire-and-forget from NSIS, so I can't see their result on the machine. Please confirm they landed with mode: fresh.
  • Item 11: the sign-in URL Hydrogen opened carries &install_id=9b88f11a-...; I saw it in the address bar. The auth side still has to read and store it.

Not triggered by this run (they need a specific action):

  • quit (close before setup finishes)
  • crash_report (the app dies mid-setup)
  • error_screen and its button (a stop card)
  • first_use (after setup completes)
  • signin_screen cancelled and timed_out
  • uninstall

Each is wired to the code path named in the commit, and the setup-pitfalls test scripts drive most of these screens on every build.

Notes on choices:

  • slow uses the expectations you suggested. For ensure-workspace as a whole I used 450 s, the sum of its three long phases; with 30 s it would flag every install.
  • first_boot runs from the end of wsl --import until ensure-workspace ends, which is when the editor answers.
  • quit is not sent for the planned WSL2 restart or for an update.
  • The installer's fail reason is failed or cancelled. NSIS cannot tell disk_full from access_denied without more work.
John Lauer · 15d ago

Follow-up: the 1.0.570 clean install on winvm (install_id 9b88f11a-0e45-4e2e-b78a-9f217501fa98) has now finished end to end.

  • Sign-in approved at 14:24:08 UTC: signin_screen and the wait end both HTTP 200.
  • Then setup_step for activate-editor, verify-setup and open-welcome, first_use (opened_editor) and setup_complete, all HTTP 200, the last at 14:24:34 UTC.

One quirk worth a look on the dashboard: a slow event for activate-editor fired at 14:24:15, seven seconds after the sign-in wait closed. That step's time counts the 12 minutes spent waiting on the human, so it trips the threshold. I'll change the slow check to subtract time spent in a wait span.

John Lauer · 12d ago

Three days of v2 data from public installs on 1.0.575. It works: the dashboard now explains most failures. What it shows, and what to fix next, most important first.

1. The import failure is still unexplained: hresult never arrives. Every wsl_import fail has exit_code -1 and an empty hresult. Example: 4685da4b-280a-4f27-8413-b9aca542bb07 (WSL 2.7.12 already installed, hypervisor running, 55 GB free). Import failed in about 1 second, and this person pressed retry and continue on the import_failed screen for 17 hours. Please capture the actual stderr of wsl --import (decoded from UTF-16) and send its Wsl/... error code or 0x8... value as hresult, and the first 120 characters of the text as error.

2. virt_fw reads "no" on machines where virtualization works. Windows reports VirtualizationFirmwareEnabled = false whenever a hypervisor is already running. c54e7b13-993c-4d08-b1bd-e320f97e1e2b says virt_fw: no, hypervisor: yes, and imported fine. The dashboard now treats hypervisor: yes as virtualization on. Please check the in-app virtualization pre-check: it should not show virtualization_off when a hypervisor is present. Machines with both values "no" really are off: f8660b15-f6bf-4a8a-a34c-f7004426d285 and 11d0d219-5e33-4dbc-8387-acd5d7200aae both saw the virtualization_off screen and pressed help, and neither came back. The help needs to get people through the BIOS change, or at least tell them to come back after it.

3. wsl --install exiting 1. 3eb6517f-3082-40a2-909f-2d8779f02985 (virtualization on, only 17 GB free) and f8660b15-... (virtualization off). Both show import_failed, when the real problem is the WSL install. Please send the stderr of wsl --install too, and show a screen specific to it. A low-disk check before we start would also help (we need room for 585 MB plus the imported disk).

4. wsl_install stays open with no end. 4121b5fc-7a1f-42f0-9ad8-c147fbee2fed and 55416aaf-57b4-47eb-b54b-ab34e0e21791 (Windows Server RDSH) started wsl_install and never sent ok or fail. Both then uninstalled. On 2a9ffed2-8eba-4743-9903-e6ad93c247fc, wsl_install took about 10 minutes and then went silent with no heartbeats (probably the restart, but no wait span of kind reboot was sent).

5. Small wire bugs:

  • image_download sends start, then either ok with ms: 0 or nothing, even when progress reached 100%. Please send ok with the real duration when the download finishes, and skip the phase entirely when the image is already cached.
  • quit arrives with an empty step and phase (4685da4b-...). It should carry the step that was on screen.
  • The installer sends extra installer start pings under new install IDs (4b74b0df-79b3-4d71-9de7-9aca0b9665ab, 67d3147d-7340-4141-be04-322fff88e413, same second as 55416aaf-...). That is probably the elevation relaunch. Please reuse one ID across the relaunch. The dashboard drops these for now.

For reference, how it is going: since v2 shipped, 38 public machines tried to install and 19 finished (one more is still going). Downloads run 5 to 480 Mbps, and nobody quit during the download. Every loss is in WSL install or import.

John Lauer · 12d ago ·edited

Next: hardware fields, so we know how hard visualizations can push. The fleet already accepts these, and they will show up in the funnel and the new hardware breakdown as soon as a build sends them.

Send them on first_launch and on the update check Hydrogen already makes (no extra calls home: they only ride along on a request the app sends anyway). The update check matters most: existing users never send first_launch again, so it is the only way they report.

field value source on Windows
cpu full CPU name, e.g. 13th Gen Intel(R) Core(TM) i7-13700H Win32_Processor.Name
cpu_threads logical processors Win32_Processor.NumberOfLogicalProcessors
gpu every display adapter, joined with ; Win32_VideoController.Name
gpu_vram_gb memory of the largest adapter, whole GB registry HardwareInformation.qwMemorySize under the display class key (AdapterRAM caps at 4 GB, so do not use it)
webgl_renderer what the Hydrogen webview actually renders with in the webview: WEBGL_debug_renderer_info then UNMASKED_RENDERER_WEBGL
webgpu yes / no in the webview: navigator.gpu?.requestAdapter() returns an adapter

webgl_renderer and webgpu matter most: a machine can have an NVIDIA card while the webview falls back to software rendering, and those two fields show what our visualizations actually get. ram_gb is already sent on first_launch; please add it to the update check as well.

John Lauer · 12d ago

Hardware fields are out in Hydrogen 1.0.593 (insiders).

cpu, cpu_threads, gpu, gpu_vram_gb, webgl_renderer, webgpu and ram_gb ride on first_launch and on the update-check ping, at most once a day. The update-check ping also sends them once more the first time the webview pair is known. Sources:

  • cpu: the registry's ProcessorNameString.
  • cpu_threads: logical processors.
  • gpu: every real DXGI adapter (software adapters excluded), joined with ; .
  • gpu_vram_gb: the largest adapter's DXGI DedicatedVideoMemory, which has no 4 GB cap unlike AdapterRAM.
  • ram_gb: installed memory.
  • webgl_renderer / webgpu: measured inside Hydrogen's own webview (UNMASKED_RENDERER_WEBGL, and navigator.gpu.requestAdapter() capped at 3 s), reported by the main window 5 s after launch. first_launch waits up to 10 s for them.

Verified on ConfRoomROG, read back from the fleet's /api/fleet:

field value
cpu 12th Gen Intel(R) Core(TM) i9-12900H
cpu_threads 20
gpu NVIDIA GeForce RTX 3070 Ti Laptop GPU; Intel(R) Iris(R) Xe Graphics
gpu_vram_gb 8
ram_gb 40
webgl_renderer ANGLE (NVIDIA, NVIDIA GeForce RTX 3070 Ti Laptop GPU (0x000024A0) Direct3D11 vs_5_0 ps_5_0, D3D11)
webgpu yes

Also in 1.0.593:

  • Uninstall ping: the uninstaller sends event: uninstall with removed_workspace, days_installed and setup_completed on a real uninstall only. The fleet lists these under "Uninstalls" on the setup view and at /api/uninstalls.
  • Recording: /recording/native/* records through Adom Bridge's recorder (#103).
John Lauer · 12d ago

The hresult landed, and it explains the #1 install failure. Install 9601e2a2 (1.0.575, Windows 11 Pro 25H2, virtualization on in firmware, no WSL before):

UTC what happened
18:47:06 wsl_install ok, wait reboot start (we asked for a restart)
18:47:21 person uninstalled and reinstalled Hydrogen instead of restarting
18:49:02 wsl_import fail, exit -1, Wsl/Service/RegisterDistro/CreateVm/HCS/HCS_E_SERVICE_NOT_AVAILABLE, then we showed virtualization_off
18:51:25 same failure, same screen
18:59:45 wait reboot end (they finally restarted)
19:00:55 wsl_import ok in 45 s

HCS_E_SERVICE_NOT_AVAILABLE means the Host Compute Service is not running yet because Windows has not restarted since the WSL/VM platform was installed. This matches the week's pattern of import failures "right after a fresh WSL install".

Fixes, in order:

  1. Never run wsl --import while a WSL-install restart is pending. If the reboot wait is open (or Windows reports a pending reboot), go straight to the restart screen, including on a fresh launch after a reinstall. The reinstall here wiped our knowledge that a restart was owed. Check HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending or the WSL install marker, not just in-memory state.
  2. Map HCS_E_SERVICE_NOT_AVAILABLE (and HCS_E_HYPERV_NOT_INSTALLED) to a "Restart Windows to finish installing WSL2" screen with a Restart button, not virtualization_off. This person has virtualization on (virt_fw: yes), and telling them otherwise is why people quit or reinstall.
  3. Show virtualization_off only when virt_fw and hypervisor are both no (see the note on virt_fw above).

Thanks for shipping the stderr: one install was enough to find this.

John Lauer · 12d ago ·edited

Hardware fields work (thank you), and they already show a problem: the webview renders on the wrong GPU. On laptops with two GPUs, Windows hands WebView2 the power-saving one:

machine GPU installed Hydrogen's webview renders on (webgl_renderer)
staff laptop RTX 4090 Laptop + Iris Xe Intel Iris Xe
public user, Brazil GTX 1650 + UHD Intel UHD
staff laptop RTX 5070 Ti + Arc 140T NVIDIA (correct)

Next round, most valuable first:

  1. Report the GPU setting the person picked. Hydrogen already has the setting (gpu.rs: Let Windows decide / High performance / Power saving), and the choice stays entirely theirs: no prompts or suggestions. Just add gpu_pref with gpu::preference() (auto, high or low) to the fields that ride along on the update check, next to webgl_renderer. Then the dashboard can show how many people chose each, and what their webview renders on as a result.
  2. Updater telemetry. Since 9 Oct, 13 machines are still on public 1.0.545, and 3 are on 1.0.201, although public is 1.0.596. We cannot tell whether the update was not offered, failed to download, or failed to apply. Send event: "update" with state: available | downloading | downloaded | applying | applied | failed, from_version, to_version, error. One failed reason across those 16 machines is probably the whole story.
  3. The sign-in states from #109 (callback_received / token_exchanged / failed), so the email sign-up fix can be confirmed from telemetry and not just by a recording.
  4. Pending-restart guard (the comment above): no wsl --import while a WSL-install restart is owed, and a "Restart to finish" screen for HCS_E_SERVICE_NOT_AVAILABLE.
  5. Health after setup, only on the update check. No new calls home: once setup is done, Hydrogen should contact us only when it already checks for a new version, and these fields ride along on that same request. Add workspace_ok (yes/no: the distro starts and code-server answers, as last observed, not a fresh probe just for us), disk_free_gb, and ai_prompts_since_last_check (a count, no content). That tells us who is still using Hydrogen a week after install, and whose workspace broke later.
John Lauer · 9d ago

Round 2 is out in Hydrogen 1.0.603 (insiders), all five items. It passed the full release test: 12 of 12 setup scenarios on the test VM, and both PC sweeps clean.

  1. gpu_pref (auto / high / low, from gpu::preference()) rides with the hardware fields on the update check, next to webgl_renderer. Like the other hardware fields, it is sent at most once a day.
  2. Updater telemetry: event: "update" with state = available, downloading, downloaded, applying, applied or failed, plus from_version, to_version and a short error code on failed. applied is sent by the new version on its first start, so a silent failure to apply shows up as applying with no applied after it.
  3. Sign-in states have been out since 1.0.596 as event: "signin_screen". The state field: shown, opened_browser, approved, token_saved (reason = backend or frontend), failed (reason = link_expired_unapproved, no_approval_before_expiry, carbon_unreachable or token_not_written), timed_out and cancelled. Hydrogen polls the sign-in instead of receiving a callback, so approved and token_saved take the place of callback_received and token_exchanged.
  4. Pending-restart guard: before wsl --import, Hydrogen checks the Host Compute Service. If it is missing or stuck, a WSL restart is still owed, so Hydrogen shows the restart screen and sends wsl_reboot: "restart-owed" instead of attempting the import. An import that still fails with HCS_E_SERVICE_NOT_AVAILABLE goes to the same screen, not virtualization_off. Verified by reproducing the exact sequence from install 9601e2a2 (uninstall and reinstall without restarting): setup asked for the restart and logged host compute service: missing (Windows reboot pending: true).
  5. Health after setup, on the update check only: workspace_ok (yes / no, last observed, no extra probe), disk_free_gb and ai_prompts_since_last_check (a count, no content). No new calls home.

One fix on the fleet side today: the fleet service keeps only the ping fields on its allowlist, and these new ones were not on it, so 1.0.602's values were being dropped on arrival. Fixed in adom/service-fleet 7e0f9b4f. A 1.0.603 machine's next update check then stored workspace_ok: yes and ai_prompts_since_last_check: 0. The update events' version fields will show when 1.0.604 reaches the test machines, which is building now.

Public is still 1.0.601, so public machines start sending these after the next promotion.

John Lauer · 9d ago

Update events confirmed end to end. Both test PCs moving 1.0.603 to 1.0.604 reported available, downloading, downloaded, applying, then applied from the new version, about a minute from start to finish. The fleet now has an insider-only GET /api/updates (adom/service-fleet 8ad59214, documented in its README) listing every updater event from the last 21 days, so the machines stuck on 1.0.545 will show which step they stop at once they run 1.0.602 or later.

John Lauer · 3d ago

Closing: everything in this issue has shipped and the fleet confirms it is arriving.

  • Round 1 (all 11 items) in 1.0.570, the hresult capture and hardware fields in 1.0.575 to 1.0.593, and round 2 (GPU preference, updater events, sign-in states, the restart-owed guard) in 1.0.603.
  • Checked today against the fleet: of the 50 install attempts since October 14, every one has an outcome with a reason. 27 complete, 11 uninstalled, 6 failed (each with its step, phase and error), 2 quit, 1 crashed, 1 stalled, 1 closed the window, 1 shut down. Updater events arrive on the update API.

New questions from the data go in new issues, so each has one clear ask.

Log in to reply.