Closed bug report

Mesh comes back without its relay after an update apply on AdomLapper; settings_set of an unrelated key bounces the mesh; CLI swallows the egress target_not_found

John Lauer · 2mo ago ·closed by John Lauer

Seen on AdomLapper (embedded under ah 1.0.286) during the 2.1.79 ship, 2026-09-08. Not a 2.1.79 code change — 2.1.79 touched only hd_bridge.rs — but it surfaced on the first restart after the apply and it took the cross-box paths down on this box for the rest of the session.

Symptom

After apply_update (2.1.78 → 2.1.79) the mesh came up without its relay: desktop_mesh_status → relayConnected:false, lastError: "Failed to connect to relay server", all three peers connected:false, path:null. ConfRoomROG and winvm, which took the same update minutes later, came back with relayConnected:true. The relay host itself answers (HTTP 200, 90 ms).

Effect: every --target <peer> from this box fell to the local relay list (self-only) → target_not_found; arav-rog (mesh-only) became unreachable; a cross-box permission card could not have mirrored here.

The bind lines tell the story (ab startup.log, local time)

[14:19:26] MESH: node … "AdomLapper" bound [0.0.0.0:55878, [::]:55880]                       ← post-apply start: DIRECT-mode bind, no relay
[14:19:30] MESH: stopped by the user (mesh_enabled = false)                                  ← 4 s later, something wrote mesh_enabled=false
[14:19:32] MESH: node … "AdomLapper" bound [0.0.0.0:53712, [::]:53713]                       ← restarted, still no relay
[14:27:54] MESH: node … "AdomLapper" bound [127.0.0.1:64827, [::1]:64829] relay http://127.0.0.1:62093/   ← a settings_set (cross_box_toasts) bounced it: RELAY mode this time
[14:41:56] MESH: node … "AdomLapper" bound [0.0.0.0:63533, [::]:63535]                       ← another settings_set bounced it: no relay again

The default mode is relay (loopback bind + shim). Three of four starts came up in the direct shape with no relay, and the one that had the relay was undone by the next unrelated settings write. Nobody touched the mesh settings; the writes were cross_box_toasts toggles for the #169 proof.

Two things to look at

  1. What wrote mesh_enabled=false 4 s after the post-apply start? Suspect: ah's re-adoption after an update (ad-supervisor → /embedded/enter → a settings push) carrying a stale mesh snapshot. If ah owns ab's settings in embedded mode, an update apply must not let a stale copy flip the mesh off/into direct mode.
  2. settings_set of an unrelated key restarts the mesh (commands.rs ~3498/3525 mesh::stop() on the settings path), and the restart does not reliably come back in the configured mode. A toast-preference write should never touch the mesh; and if the mesh must restart, it must re-read mesh_mode/mesh_relay_url from the same config it just persisted.

Also seen while looking: a foreign PID (13388, not an adom-bridge.exe in tasklist) holds 127.0.0.1:24682, the OLD fixed shim port from before the ephemeral-port change; harmless now (the shim binds :0) but worth knowing it's there.

With the mesh down, --target <peer> hands the call to the local ab's egress (cli/src/relay.rs direct_call_target) and returns whatever JSON comes back as the result — including {"error":"target_not_found"} — instead of treating it as a miss and falling through to send_and_wait on the relay. That is why a directed notify_user "failed" while other verbs appeared to work, both today and at the #167 close. Fix: an egress payload with error ∈ {target_not_found, ambiguous_target} should Err so the relay path runs.

2 Replies

John Lauer · 1mo ago

Reproduced the third symptom cleanly today, and it is narrower and more concrete than "the CLI swallows it".

On ConfRoomROG, same box, same minute, ab 2.1.114:

targets                ->  ConfRoomROG  connected=True  via=relay
desktop_list_windows   ->  ok            via=mesh
ping                   ->  connected
notify_user            ->  target_not_found     (twice, reproducibly)

So it is not that the box is unreachable. A plain desktop verb reached it over the mesh in the same breath that notify_user declared the target missing.

The cause is that notify_user is a different message type, not a command, so it never got the mesh-first egress that plain verbs have had since 2.1.40 and that 2.1.97 extended to the file verbs. It is relay-only, and the relay's client list said connected when the relay leg was not actually usable. Two separate faults stacked: the transport gap, and targets reporting a stale relay row as live.

Practical effect: cross-box toasts silently fail to any box whose relay leg is flapping, while every other verb to that same box works, so it reads as "toasts are broken" rather than "notify has no mesh path".

Not fixed in 2.1.115. Filing it here rather than opening a new issue since it is this issue's third symptom, now isolated. The fix is to route notify through direct_api_peer_egress like commands and file transfers, and to stop targets presenting a relay row as connected when the relay leg has not answered.

John Lauer · 1mo ago

Closing as not reproducible on the current insiders build, with evidence; reopen if it shows again.

AdomLapper took four auto-update applies today (2.1.132 → 2.1.133 → 2.1.134 → 2.1.135 → 2.1.136, each through the real staged-NSIS path with the embedded restart). After the last one, read-only:

desktop_mesh_status (AdomLapper, 2.1.136):
  enabled true, mode relay, relay https://meshrelay-o6vza2ooz7ou.adom.cloud/, relayConnected true, relayAuth token

So the mesh came back WITH its relay after each apply. The settings_set half of the report (an unrelated key restarting the mesh) is covered by the 2.1.61 mode-change rule: a settings_set that does not change mesh_mode/mesh_enabled/mesh_relay_url no longer stops or rebinds the endpoint. If a box ever comes back relay-less again, startup.log will have the MESH: bind line and desktop_mesh_status.relayUpstreamStatus names the relay's HTTP answer; post those and I will reopen.

John Lauer closed this · 1mo ago

Log in to reply.