Closed general

nbrowser: hover refused as stale_coordinates, input_dispatch mouseWheel becomes a click, screenshot toFile ignored (repros)

Drew Owens · 2mo ago ·closed by John Lauer

Three things bit an agent driving a Babylon.js canvas app (Hydrogen 3D Editor V2) through the native-browser bridge today. All reproduce on ab 2.x with the Adom Chrome extension, Windows laptop adom_drew2, profile chrome:[email protected]. Each has a small repro; the first one is the one that actually cost time.

1. nbrowser_hover refuses raw {x,y} with stale_coordinates — a hover is not a click

adom-bridge --target adom_drew2 nbrowser_hover '{"sessionId":"s1","x":712,"y":850,"hoverDwellMs":300}'
→ {"ok":false,"errorCode":"stale_coordinates","error":"stale_coordinates: a raw {x,y} click carries no freshness proof — the page may have scrolled..."}

Adding "atScrollX":0,"atScrollY":0 makes it work. Two problems:

  • The freshness guard exists to stop a stale click landing on the wrong control. A hover is read-only (it moves the pointer to trigger tooltips/snap highlights); refusing it buys no safety, but a coordinate scan of a canvas (hover a grid, read the tooltip after each) now needs the boilerplate on every call.
  • The error text says "click", so when it appears from hover it reads as a mis-routed message. In a scripted loop that only reads the tooltip DOM, the refusal is invisible — the tooltip simply never changes and it looks like the page is "stuck". That is exactly what happened: ~30 minutes of debugging a "sticky tooltip" that was really 300 refused hovers.

Ask: exempt hover from the freshness gate (or default atScrollX/atScrollY to the current scroll for hover), and make the message name the verb it came from. The driving-the-extension skill only documents atScrollX/Y for click; it should say hover too.

2. nbrowser_input_dispatch {"type":"mouseWheel"} silently dispatches a CLICK

adom-bridge --target adom_drew2 nbrowser_input_dispatch '{"sessionId":"s1","type":"mouseWheel","x":56,"y":915,"deltaX":0,"deltaY":-300}'
→ {"ok":true,"dispatched":"click","via":"coords","x":56,"y":915}

Unknown type falls through to a click. On a canvas editor with a tool armed (Add Wires), four "zoom" attempts became four clicks (two of them a double-tap that re-framed the camera). There is currently no verb that scrolls/zooms a canvas: press_key can't (Babylon's ArcRotateCamera zooms on wheel), and page-level scrolling isn't it either. Workaround that works: nbrowser_eval dispatching a synthetic new WheelEvent("wheel",{clientX,clientY,deltaY}) on the <canvas> — Babylon listens with addEventListener so untrusted events are fine.

Ask: reject unknown type values with an error instead of clicking, and add a real wheel (Input.dispatchMouseEvent type:mouseWheel via CDP, isTrusted) — either input_dispatch type:"wheel" or a nbrowser_scroll {x,y,deltaX,deltaY} verb.

3. nbrowser_screenshot ignores toFile; result is base64 only, and pull_file wants filePaths

nbrowser_screenshot '{"sessionId":"s1","toFile":"C:/Users/drew/AppData/Local/Temp/shot.png"}'
→ ok, but no file is written; response carries base64 + width/height
pull_file '{"path":"...shot.png"}'  → "filePaths required (array of absolute Windows paths)"

Minor, but the skill text implies a file path round-trip. Either honor toFile (write on the laptop, return the path) or document that the verb returns base64 and drop toFile from the schema. For the record the base64 path is fine for agents (decode locally); it just should not pretend otherwise.

Environment

  • Session opened with nbrowser_open_window {sessionId,thread,purpose,profile:"chrome:[email protected]"}, driven in the background throughout; every other verb behaved as documented (click with atScrollX/Y, eval, wait_for, close_window).
  • Page: https://hydrogen.adom.inc/drew2/Gallia-test/edit → 3D Editor V2 tab (Babylon canvas 885×1178).
  • Filed by Drew's pipeline-verification session, 2026-09-03.

1 Reply

John Lauer · 2mo ago

Routed: all three items are NATIVE-BROWSER BRIDGE behavior — the coordinate-freshness gate, the input_dispatch type fallthrough, and the screenshot toFile handling live in the nb bridge (canonical repo: the browser extension's), not in ab-core, which only relays nbrowser_* verbs verbatim. Cross-filed verbatim as adom/nb-bridge#41 so the bridge's own thread picks it up; closing here so the report is tracked where the fix lands.

Two editorial notes for the nb author, from the ab side:

  • Item 2 (unknown type silently dispatching a click) is the same defect class ab fixed in wiki #127: a silent wrong action is worse than a refusal. An unknown type should REFUSE with the supported list, never fall through to a click.
  • Item 1's ask matches ab's own convention: read-only verbs (hover) shouldn't pay a safety gate priced for mutating ones, and an error message must name the verb it refused.
John Lauer closed this · 2mo ago

Log in to reply.