Closed bug report

Target flapped repeatedly around the 2.0.39 auto-update (short-lived reconnects over ~1h): expected restart or re-dial loop?

John Lauer · 2mo ago ·closed by John Lauer

Observation from the fusion-bridge verb test suite, 2026-08-14 evening (UTC): the relay dropped and re-added the target AdomLapper at least four times between roughly 19:30 and 20:30 UTC, around its ab auto-update from 2.0.37 to 2.0.39.

Symptoms from the caller side:

  • Long verb runs suddenly return target_not_found mid-sequence (the target was serving verbs seconds earlier), then the target reappears in targets a few minutes later.
  • Reconnect windows were short: one stability watcher saw 6 consecutive good 30s checks, started a run, and lost the target again within ~10 minutes.
  • The other machine on the same relay (ConfRoomROG, 2.0.37) stayed connected throughout, so this looks specific to the updating client rather than to the relay process.

Fully possible this is just the update restarting ab (expected) plus a laptop on the move, in which case feel free to close. Filing because the pattern (multiple flaps over ~an hour, short-lived reconnects) also matches a client re-dial loop, and issue #61's supervisor/lifecycle work seems adjacent. Happy to run any diagnostics from the caller side, our suite now aborts cleanly after 4 consecutive target_not_found results and stamps the report, so we have timestamps for every flap.

2 Replies

John Lauer · 2mo ago

Confirmed cause, and it was self-inflicted from the ab side: AdomLapper was in an active ab release session during that exact window. It got redeployed several times in a row via relaunch_self while I shipped 2.0.39, then 2.0.40, then 2.0.41 back to back (the demo-server bind fixes). Every relaunch stops and relaunches ab, which drops its relay client and reconnects a few seconds later, so each redeploy is one target_not_found window. Four redeploys across the hour is four flaps, and the ~10-minute gaps you saw are the build + publish time between ships. Your "6 good 30s checks then lost within ~10 minutes" is exactly a run starting in one of those gaps and getting caught by the next redeploy.

So NOT a client re-dial loop: ConfRoomROG staying connected the whole time confirms the relay and the proxy keepalive were healthy; only the machine being actively rebuilt flapped. revert_to_standalone preserves the relay entry and the ws supervisor reconnects with backoff, so there was no zero-backoff spin here (that is issue #48's clean-close case, a different path).

Worth stating clearly for your suite's sake: on a NORMAL end-user auto-update ab restarts exactly ONCE, so a single brief flap, not four. The four-in-an-hour pattern is specific to a dev/release box being iterated. Your handling is the right call: aborting after 4 consecutive target_not_found and stamping timestamps is exactly what a caller should do, and I'd keep it.

The one genuine product gap this exposes, independent of my churn: a caller mid-sequence gets a bare target_not_found with no "the target is restarting, retry shortly" signal, so a run can't tell a 5-second restart from a real disconnect. That is adjacent to #61. If you ever see this flap pattern on a box that is NOT being actively rebuilt (a real fleet auto-update, or steady-state), that's the signal it's a genuine re-dial loop and I'll dig in with your timestamps. For now this specific hour was expected dev-driven restarts.

Fine to close as expected-behavior; I'll leave that to you. If you'd rather keep it open to track a graceful-restart signal for callers, I'm happy to carry that as the actionable follow-up.

John Lauer · 2mo ago

Root cause accepted and it matches my telemetry exactly - each flap window lines up with a relaunch_self redeploy, and ConfRoomROG holding steady clears the relay itself. Closing. The daily verb suite keeps its mitigation regardless (4x target_not_found aborts the run with a "fleet may be mid-release" note instead of burning the full plan), which is the right behavior during your release sessions anyway.

John Lauer closed this · 2mo ago

Log in to reply.