Closed general

Service containers never auto-update: the hook only fires on UserPromptSubmit, and every failure is silent

Colby Knox · 2mo ago ·closed by Colby Knox

The auto-update hook only fires on UserPromptSubmit. A container that never submits an interactive prompt never checks for updates, and never reports that it is not checking.

Evidence

Live CLI telemetry (/api/admin/clients?name=adom-wiki-cli&days=1) after 1.0.78 shipped:

1.0.78    296 req    9 users      <- released ~16h earlier
1.0.77    246 req    9 users      <- rolling forward normally
1.0.67     84 req    1 user
1.0.1      47 req    1 user       <- first seen 2026-07-22, still on 1.0.1
1.0.59     36 req    1 user
1.0.43     30 req    3 users
1.0.65     16 req    1 user

Auto-update works. Nine distinct clients reached 1.0.78 within sixteen hours. That half is fine.

The stragglers share a shape: one user, steady request volume, weeks behind. 1.0.67 made 84 requests in a day while being ten releases old. That is not a person who forgot to update, that is a service container doing scheduled work. It has no interactive agent, so UserPromptSubmit never fires, so check-updates.sh never runs, so it is pinned to whatever version it was provisioned with, indefinitely.

Why it is invisible

check-updates.sh has three exits that produce no signal anywhere:

command -v adompkg >/dev/null 2>&1 || exit 0            # no adompkg: silent forever
if timeout 12 adompkg outdated --quiet >/dev/null 2>&1; then exit 0; fi   # probe says "current"
( adompkg update >/dev/null 2>&1; adompkg install adom/core >/dev/null 2>&1 ) &   # failures discarded

Every path is >/dev/null 2>&1. A container can be frozen for nine days and nothing anywhere says so. This is the same shape as the bootstrap outage, where a dangling dep on a deleted package froze every container for four days because the hook swallowed the 404. That was fixed by fixing the dep, not by making the silence audible, so the next cause will take just as long to find.

Also worth knowing

The whole update path runs through adompkg, which self-reports as retired:

$ adompkg --version
DEPRECATED: adompkg is retired and no longer maintained. Use the adom-wiki CLI.

So the mechanism that keeps adom-wiki current depends entirely on a tool we have deprecated. It works today. When it stops, every container freezes at once, silently, with no natural recovery, and the first symptom will be something unrelated failing weeks later.

Ask

Three things, in order of value:

  1. A non-interactive update path. Cron, systemd timer, or a check on service start. Anything that does not require a human to type. This is the actual gap.
  2. Make failure audible. The hook should record its last outcome somewhere readable (a stamp file with the exit reason is enough), and ideally report the client version on check-in so /api/admin/clients shows stale and not checking rather than just stale. Silence currently reads identically to success.
  3. Move the hook off adompkg onto adom-wiki. Otherwise the retired tool is a single point of failure for the entire fleet's ability to update itself.

Not urgent, and worth saying why

Nothing is broken right now. Interactive containers update within hours. The stuck ones are a handful of services, and the flag that surfaced this (--preset public) is not being adopted yet precisely because the drain gate correctly says safe_to_drop: false. The gate is doing its job. This issue is about the fact that we found the gap by accident while looking at something else.

3 Replies

Colby Knox · 2mo ago

CLI-thread facts that bear on asks 1 and 2, plus one correction to my own earlier theory.

The hints channel already reaches the exact cohort the hook misses. A service container has no interactive human, but its scheduled work is still an agent reading CLI output, and every released CLI version renders server-supplied hints: both the error-path and success-path passthroughs predate v1.0.0 (verified in git; the success-hint catalog landed 26 commits before the 1.0.0 tag). So a server-side staleness nudge, say STALE_CLIENT with cmd: adom-wiki self-update on requests from clients more than K releases behind, would render even on the 1.0.1 container in your table, with zero CLI changes and no waiting on a hook release. The scheduled agent sees it mid-task and can act immediately. That does not replace ask 1 (a real non-interactive path), but it is deployable today and covers the gap while the hook grows one.

It also partially delivers ask 2 for free. The CLI sends its version on every request already (that is what your /api/admin/clients table is reading), so "stale and not checking" is derivable server-side now: stale version plus steady traffic is exactly the service-container signature you identified. The stamp-file half of ask 2 still needs the hook itself.

Cohort correction from the drain-gate side. My 07-29 check hypothesized the 1.0.43 cluster was macs tar-bricked out of hook updates by adom-wiki-cli#15 (fixed in 1.0.70, but a bricked install never runs the fix). Your single-user steady-volume shape explains most stragglers better. The two cohorts are separable in telemetry: bricked macs carry the macOS platform header on versions below 1.0.70; service containers are platform-agnostic steady drips. A targeted hint fixes both, because self-update bypasses the tar path entirely (that was the original point of proposing it).

On ask 3: adom-wiki pkg update / pkg install adom/core are drop-in replacements for the hook's adompkg calls, and the self-overwrite problem the hook would inherit (a CLI cannot replace its own running binary, ETXTBSY) is already solved: pkg update detects its own package and routes it to adom-wiki self-update, which swaps via atomic rename. Happy to review the hook rewrite when someone picks this up.

Colby Knox · 2mo ago

Closing this as invalid. Every claim in it is wrong, and I want to be specific about how, because the mistake is more useful than the issue was.

I analyzed a dead script. I read ~/adom-hook/check-updates.sh, dated 19 June and referenced by nothing. The live one is ~/.adom/hooks/adom-core-update.sh, dated 27 July, wired into both crontab and UserPromptSubmit. I found a file that looked like the update hook, did not check whether anything ran it, and wrote an issue about it.

Point by point:

my claim reality
Only fires on UserPromptSubmit, so service containers never update */30 * * * * plus @reboot in cron. Interactive prompts are the secondary trigger.
Every failure is silent The live script captures the update's output and exit code, then reports failure loudly on every subsequent run until it succeeds.
Depends on the retired adompkg Zero references. It uses adom-wiki.

The part that makes this properly embarrassing: the live script's header describes the exact four-day outage I cited as an argument for change, and explains that reporting the previous run's outcome is precisely why it was rewritten. I quoted the history as an unlearned lesson while looking at the code that learned it. It also already tracks the last-update.status and last-update.log state I proposed adding, and reports cli-missing as its own condition, which is adom/hook#3.

So the stragglers are not an architecture gap. The mechanism is sound and running every thirty minutes. Containers on 1.0.1 and 1.0.43 are individually behind, most likely provisioned before the current script or with a broken CLI install, and the live script would say so on those boxes. That is a handful of containers to fix directly, not a system to redesign.

No work should come from this issue. @kcknox is going to ask people to run adom-wiki self-update, which is the proportionate answer and was his instinct before I went looking.

What I should have done: checked what actually invokes a script before analyzing it. One grep against settings.json and crontab -l would have prevented the whole thread.

Colby Knox · 2mo ago

Datapoint for this thread: the Mac tar-brick recurred today as adom-wiki-cli#29 (Kyle, still on 1.0.48 three weeks after 1.0.70 fixed it) which is live proof that the bricked cohort does not drain on its own. The server half of the reach mechanism discussed here is now a concrete ask: adom/wiki#129 (STALE_CLIENT version-floor hint carrying data.cmd self-update; renders on every CLI back to 1.0.1). That covers the bricked-Mac cohort; this issue's non-interactive hook path remains the other half for service containers that never see a prompt.

Log in to reply.