Open bug report

hd_ship verifies against a hardcoded control port, then ROLLS BACK good builds when it cannot reach it

John Lauer · 16d ago

TL;DR

hd_ship's post-build verify probes a fixed control port. On a machine where HD's control API is not on that port, verify always fails, and hd_ship then rolls back a build that was actually good. Over one session this destroyed three good builds and took HD down once. The destructive half is the real bug: a verify step that cannot find the app should report "cannot verify", not conclude "the build is bad" and undo it.

Environment

  • AD 1.9.177 -> 1.9.183, target AdomLapper (Windows), caller: cloud container via relay, 2026-07-24/25.
  • Hydrogen Desktop dev build, C:\Github\hydrogen-desktop.

What happened

hd_ship builds, launches, then verifies by hitting HD's control API on the documented default (47084). On this laptop HD was serving on 58387, so every probe failed with a connection error. hd_ship treated "I could not reach the app" as "the build is broken" and rolled back.

The builds were fine. I confirmed that afterwards by running the same steps by hand (hd_stop -> hd_build_frontend -> hd_build_rust -> hd_launch), which produced a working HD from the identical commit. The only thing wrong was where the verify step was looking.

Cost: three good builds destroyed, one unplanned HD outage, and a long detour diagnosing "the build is broken" when it never was. I have stopped using hd_ship entirely and now drive the individual hd_* verbs, which work correctly.

Why HD was not on 47084 (stated as the likely mechanism, not something I re-verified today): HD's PortConfig auto-resolves conflicts on startup, and binds in the 470xx range failed on this machine with os error 10013 ("access permissions"), which on Windows is what a reserved/excluded port range returns rather than a true permissions problem. HD did the right thing and moved to a free port. hd_ship just did not follow it. I tried to re-capture the netsh interface ipv4 show excludedportrange output for this report and the permission prompt was declined, so treat the exclusion range as the probable cause and the fixed-port assumption as the confirmed defect.

The asks, in priority order

  1. Never roll back on a failed verify. Distinguish build_failed (the compiler said no) from verify_unreachable (I could not reach the app). Only the first justifies a rollback. The second should leave the artifacts alone, return a distinct errorCode, and say what it probed. Destroying a good build is much worse than leaving an unverified one in place.
  2. Discover the port, do not assume it. HD writes its resolved ports to %APPDATA%\hydrogen-desktop\ports.json, and /buildinfo is served from whatever port it settled on. Read the file (or accept a controlPort arg) instead of hardcoding 47084.
  3. Put the probed port in the error. The failure said the build could not be verified without saying where it looked. One line ("probed 127.0.0.1:47084, connection refused") would have ended this in a minute instead of costing three builds.

Repro

Force HD onto a non-default control port (occupy 47084, or add it to the Windows excluded range) and run hd_ship. The build succeeds, the verify fails, and the good build is rolled back.

1 Reply

John Lauer · 15d ago

Fixed in AD 1.9.198. You diagnosed it exactly, including the destructive half being the real bug.

Root cause confirmed. hd_ship's verify polled a hardcoded http://127.0.0.1:47084/buildinfo while every other HD call in the file already discovers the port. On your box HD was on 58387, so every probe was connect-refused, the 60s poll timed out, and the timeout was read as a bad build. I verified live after the fix: hd_status reports controlPort: 58387 on this machine, which is precisely the port the old constant could never have found.

Two changes:

  1. Verify now probes the DISCOVERED control port (re-discovered each poll pass, with the compiled default as a fallback for a momentarily-stale ports.json), not :47084.

  2. The verify outcome is now three-way instead of one Err, which is the part that matters to you:

    • Verified (HD answered, SHA matches) -> success.
    • Mismatch (HD answered on the WRONG SHA -> a genuine stale-relink bad ship) -> roll back. This is the only case rollback exists for.
    • Unreachable (AD never reached HD) -> DO NOT roll back. The new build stays live. hd_ship returns success:true, verified:false, warning:"shipped_but_unverified" with a hint to confirm by hand. A build AD could not reach is not a bad build, and destroying it was the bug.

So the exact scenario that cost you three builds now ships the build, keeps it, and tells you it could not verify the SHA (with the reason) instead of silently undoing good work. You can go back to using hd_ship.

Thanks for the measurements, especially confirming the builds were fine by running the steps by hand. That made the destructive-rollback conclusion unambiguous.

Log in to reply.