Open general

server_remove severs the channel carrying the command, so the paired server_add never lands (no atomic token update)

Drew Owens · 12d ago

Version: AD 1.9.213 both ends. Companion to the server_add/authToken issue. Impact: strands a container from every desktop mid-operation. Recoverable only via a second relay.

What happened

Because server_add does not update authToken on an existing entry, the only working way to change a token is server_remove + server_add. For the relay you are working through, that is a self-severing operation:

adom-desktop --target D server_remove {"name":"AdomPNP"}   # ok — and this kills my own channel
adom-desktop --target D server_add    {…new token…}        # never arrives: ok=None

server_remove deletes the entry, the WebSocket carrying the command drops, and the follow-up server_add cannot be delivered. Both desktops were left with no entry at all for that relay — not a stale one, none.

I recovered by SSHing into a different relay's container and re-adding the entry from there. Without a second relay this needs RDP or physical access to the machine — on a production machine-controller PC that is a serious outcome for a routine credential rotation.

Why it is not just "user error"

  • The docs recommend rotating tokens (#33) but there is no safe primitive to do it with.
  • server_remove cannot know it is about to cut its own transport, but AD can: it knows which server entry the current command arrived on.

Asks

  1. server_set_token {name, authToken} — atomic, no delete, no reconnect gap. This is the real fix and also fixes the companion issue.
  2. Failing that, have server_remove refuse by default when name is the entry the current command arrived through, with an explicit --force/allowSelfSever:true override and a hint naming the consequence.
  3. Consider a server_rotate_token that stages the new token, waits for a successful auth, and rolls back on failure — the pattern SSH/iptables use for exactly this class of self-lockout.

1 Reply

John Lauer · 9d ago

Addressed in AD 1.9.215 (53ee330) via the companion fix on #386: rotation no longer needs server_remove + server_add at all.

The flow is now: update the cloud profile key (desktop.container.*, same keys register_container writes) → adom-desktop server_sync --target → the desktop cloud-wins-merges the entry in place (authToken included) and reconnects. The refresh command travels over the OLD connection, the merge happens atomically in config, and the reconnect uses the new token — worst case is a few seconds of re-dial, never a stranded machine. Desktops also re-sync hourly on their own, so even a missed server_sync self-heals.

Your ask #2 (server_remove refusing to sever the transport carrying the command, without an explicit force) is a good defense-in-depth idea and is on the list, but it's no longer the load-bearing fix — with server_sync there is no reason for a rotation runbook to touch server_remove at all.

Log in to reply.