Setup telemetry on macOS, like Windows: report every first-run step so a stuck Mac install is visible (and so we can split our time from the user's)

Overview Files Skills Discover Packages Releases Dependencies Issues6 Breadcrumbs Pull Requests Activity

Closed feature request

John Lauer · 1mo ago ·closed by Kyle Bergstedt

Kyle, Hydrogen for Windows now reports every step of first-run setup, and it has changed what we can see about new users. I would like the same on macOS, because right now a Mac install is invisible from the moment someone downloads the DMG until its first update check after setup succeeds. A Mac user who gets stuck creating the Lima VM looks exactly like someone who never downloaded at all.

What Windows gives us today, a real example

A new outside user signed up on 2026-10-05 from a Windows laptop in Turkey. Because of the setup telemetry we could watch the whole install as it happened:

Time (UTC)	Event
12:13:52	first launch, Hydrogen 1.0.543
12:14:55	admin prompt shown; approved in 2 seconds
12:15:31	no WSL on the machine, so Hydrogen installed it and asked for a restart
12:16:41	Windows back up, 70 s later
12:17:53	they logged in; Hydrogen relaunched itself and resumed
12:18:57	workspace built
12:18:59	the next four steps, 1 second combined
12:19:39	signed in to Adom, 41 s after being asked
12:19:53	setup complete

From those events the dashboard splits the 6 minute wall clock exactly: 2m 57s of our code, 1m 54s waiting on the person (admin prompt, logging back in, signing in), and 1m 10s of the OS restarting. When an install dies, the same data says which step, with what error, after how long, and whether a person was the blocker. Across the last two weeks it has already found a sign-in screen nobody got past, a restart that never came back, and a 38 minute workspace download on a slow connection in India. None of that is visible for Mac users today.

What to send from the Mac app

The full Windows spec and its verification runs are on adom/hydrogen#87. Same wire shape, so the dashboard needs no Mac-specific code. Pings go to the fleet_url already carried by the public macOS manifest (you shipped that in #4), fire and forget, one ordered queue, short timeout, no retries, never blocking setup.

install_id: a random UUIDv4 created on first run and kept in the app's support directory. Send it on every ping, including the existing update-check ping. It stitches the install together before anyone has signed in.
event:
- first_launch: once, before setup begins. Skip it when setup history already exists, so an upgrade is not mistaken for a new user (Windows 1.0.512 had that bug).
- setup_step: step is your cascade's own step id, state is start, then ok or fail with ms, and on fail a short error code (the step plus the first useful code, never a stack trace or a path).
- setup_complete: once, with ms for the whole run, plus work_ms and wait_ms.
- signin: when the login first names a user.
wait spans, state start and end with ms, for anything setup cannot pass without the person. These are what make "our code versus waiting on the user" exact. On Windows the kinds are signin, uac, reboot, logon. The macOS equivalents I would expect:
- signin: waiting for the Adom sign-in.
- auth: an admin password or Touch ID prompt, if Lima or a helper needs one.
- permission: a macOS consent dialog setup is blocked on (network, local network, files), with result approved or denied.
- reboot and logon only if setup ever needs a restart on a Mac. A span is only real blocking: if setup is still doing work in parallel, it is not a wait.
Product stays hydrogen-macos, os stays macos, and nothing new identifying goes in: no tokens, emails, paths or hardware ids.

The step that matters most on a Mac is the Lima VM: image download, VM creation, first boot. Please give the download its own step if you can, since slow connections are where Windows users suffered most.

How to verify

A fresh install on a Mac with no existing Lima instance, signing in after a deliberate 2 minute wait. The dashboard should show every step, then about 2 minutes waiting on the user.
Block the network partway through the image download. The dashboard should show a fail at that step with a sensible error code.
Post both install_ids here and the adom/insiders thread will confirm the rows on the dashboard match.

4 Replies

John Lauer · 1mo ago ·edited

Hardware fields, for the same reason as on Windows (adom/hydrogen#102): we need to know how hard visualizations can push. Please send these on first launch and on the update check the app already makes (no extra calls home just for this):

field	value	source on macOS
`cpu`	chip name, e.g. `Apple M3 Pro` or `Intel(R) Core(TM) i7-9750H`	`sysctl -n machdep.cpu.brand_string`
`cpu_threads`	logical CPUs	`sysctl -n hw.logicalcpu`
`ram_gb`	whole GB	`sysctl -n hw.memsize`
`gpu`	GPU name(s), joined with `;`	`system_profiler SPDisplaysDataType` (chipset model)
`webgl_renderer`	what the app's webview renders with	in the webview: `WEBGL_debug_renderer_info` then `UNMASKED_RENDERER_WEBGL`
`webgpu`	`yes` / `no`	in the webview: `navigator.gpu?.requestAdapter()` returns an adapter

All values are strings of at most 120 characters, the same as the other ping fields. The fleet already accepts them.

John Lauer · 1mo ago

Smallest change with the biggest payoff: ping on launch and right after sign-in. Today the Mac app pings only on its update-check timer (median gap 4 hours), so a new Mac user is invisible for hours after signing up. Today a Windows user's "got it running" post went out 3 minutes after their sign-up post; a Mac user who signed up yesterday was only seen 20 hours later.

Please send the usual ping (same body as the update check) at two more moments:

on every app launch, as soon as the app starts, before anything else (event: "first_launch" the very first time, otherwise no event),
right after sign-in completes (event: "signin", with username).

That alone lets the sign-up follow-up post say "got Hydrogen running on Mac" within minutes, and the full setup steps from this issue can come after.

John Lauer · 1mo ago ·edited

One checklist, so nothing is scattered across the comments above. This is everything that makes the Mac app report like Windows Hydrogen does today. Windows has had it for a week, and it already found the two biggest install failures (a pending restart treated as "virtualization off", and new email sign-ups never reaching Approve). Ranked by value:

Ping on launch and right after sign-in (comment above). Today a Mac user is invisible for up to 4 hours. This one change lets the User Signups chat post "got Hydrogen running on Mac" within minutes. event: "first_launch" the first time, event: "signin" with username after sign-in.
install_id on every ping (a UUID made at first launch, kept in Application Support), so every event from one attempt joins up.
Setup steps: event: "setup_step", step, state: start | ok | fail, ms, error, for each step of the Mac cascade (Lima or VM install, image download, VM start, code-server, editor). Then event: "setup_complete" with ms. Use whatever step names the Mac cascade actually has, and send them in order. The dashboard learns the order from the pings.
Waits: event: "wait", kind: signin | permission | restart, state: start | end, ms. These separate "our code is slow" from "waiting on the person" (for example a macOS permission prompt or a Lima install that needs approval).
Failure detail: on fail, the tool's real error text (first 120 characters) and exit code. On Windows, this one field explained the #1 failure in a single install.
Hardware (comment above): cpu, cpu_threads, ram_gb, gpu, webgl_renderer, webgpu, on first launch and on the update-check request the app already makes (no extra calls home for this).
Download progress and heartbeat, during setup only: event: "progress" every 10% of the image download, with pct and bps; event: "heartbeat" every 60 s while setup runs; event: "quit" if the window closes before setup finishes.

All fields are strings of at most 120 characters, sent to the fleet_url from the manifest exactly like today's ping. The fleet already accepts every field above, so each item shows up on the dashboard the moment a build sends it. Item 1 alone is a small change and worth shipping first, on its own.

Kyle Bergstedt closed this · 4d ago

Kyle Bergstedt reopened this · 4d ago

Kyle Bergstedt · 4d ago

fixed: setup_step, setup_phase, wait, setup_complete, install_id and hardware on first launch since 0.1.427. 0.1.429-dev.12 adds hardware plus webgl_renderer and webgpu to the update-check ping, progress every 10% of the image download, a 60 s heartbeat and quit during setup. errors stay short codes (step + cause + exit code) so no paths or commands leave the mac

Kyle Bergstedt closed this · 4d ago