Open general

server_add: documented upsert does NOT replace authToken on an existing entry (silent auth-fail loop)

Drew Owens · 12d ago

Version: relay + desktops on AD 1.9.213, container CLI 1.9.213. Impact: high for anyone rotating relay tokens — the failure is silent and looks like an unstable relay.

What happened

Rotating the relay auth token off the well-known default (per issue #33) across 4 relays and 2 desktops.

server_add's own help says it "Upserts by name: if a server with the same name exists…", and documents authToken as "Empty string means 'keep existing token if this is an update; otherwise use default'" — which strongly implies a non-empty value does update it.

It does not. Calling server_add with the same name/url and a new non-empty authToken returns ok:true, but the stored entry keeps the old token.

Symptom (why this is nasty)

The desktop then flaps against a relay that has already rotated:

[…] WS connected: 1cf9aefd… from 127.0.0.1:54964
[…] ← 1cf9aefd… type=auth id=5005a756…
[…] Auth FAILED: 1cf9aefd…
[…] WS disconnected: 1cf9aefd…

server_list cycles reconnecting → connected → disconnected. Because server_add reported success, every diagnostic points at the relay or the network. We burned most of a rotation window on this before testing the assumption.

server_list also does not return authToken (correct, it is a secret), so there is no way to observe that the stored token is stale — the bug is unobservable through the API.

Repro

  1. server_list → note an entry X, connected.
  2. Rotate relay X's token: restart with ADOM_AUTH_TOKEN=<new>.
  3. server_add {name:"X", url:"<same>", authToken:"<new>", autoConnect:true}ok:true.
  4. Observe Auth FAILED on the relay and flapping in server_list.
  5. Now server_remove {name:"X"} then the identical server_add → connects and stays.

Workaround

server_remove + server_add. See also the companion issue about that being unsafe for the entry carrying your own command.

Asks

  1. Make a non-empty authToken actually update an existing entry, or
  2. if upsert is intentionally add-only for secrets, say so in the verb help and return something other than a bare ok:true (e.g. updated:false, tokenChanged:false) so a caller can tell.
  3. Ideally add a dedicated server_set_token {name, authToken} that is atomic and does not require deleting the entry.

1 Reply

John Lauer · 9d ago

Shipped in AD 1.9.215 (commit 53ee330) — but by fixing the MODEL rather than the upsert.

The server list's source of truth is the signed-in user's cloud profile (the desktop.container.* keys that register_container writes), not the local config file. The gap was that a LIVE desktop only re-read that profile at boot and at interactive sign-in — which is why you were forced onto the local server_add path at all.

New in 1.9.215:

  • server_sync verb (relay/CLI/direct API): makes the desktop re-pull desktop.container.* NOW and cloud-wins-merge it — the WHOLE entry is replaced, authToken included. Returns {serverCount}.
  • A signed-in AD also re-syncs hourly on its own.

So the supported rotation flow is: update the cloud key with the new token (register_container writes these same keys) → adom-desktop server_sync --target → verify with server_list. No remove/re-add, no reconnect gap beyond the re-dial, and the stale-token-invisible problem goes away because the profile is authoritative.

On the literal bug you found: server_add's authToken upsert really is add-only and its help really does imply otherwise — that help text will be corrected to say so and point at the cloud flow. We're deliberately NOT making server_add a token-rotation surface; the cloud profile is the one writer.

Log in to reply.