app

NB - Native Browser Bridge

Public Made by Adom

by adom

Your real browser, driven by AI.

nbnbeabbridgeadom-bridgenb-bridgenative-browserbrowserextensionchromeedgenbrowserpupnative-messagingcredentialschip-fetcher
Overview Files Skills Discover Packages Releases Dependencies Issues7 Breadcrumbs Pull Requests1 Activity
Closed general

hover refused as stale_coordinates (a hover is not a click), input_dispatch mouseWheel silently becomes a click, screenshot toFile ignored - repros from adom/adom-bridge#135

John Lauer · 2mo ago ·closed by John Lauer

(Cross-filed from adom/adom-bridge#135 by the ab thread - all three items are native-browser bridge behavior (the stale_coordinates gate, the input_dispatch unknown-type fallthrough, and screenshot toFile), so the fix lands in this repo, not ab-core. Original report verbatim below; the ab-side issue is closed pointing here.)

Three things bit an agent driving a Babylon.js canvas app (Hydrogen 3D Editor V2) through the native-browser bridge today. All reproduce on ab 2.x with the Adom Chrome extension, Windows laptop adom_drew2, profile chrome:[email protected]. Each has a small repro; the first one is the one that actually cost time.

1. nbrowser_hover refuses raw {x,y} with stale_coordinates — a hover is not a click

adom-bridge --target adom_drew2 nbrowser_hover '{"sessionId":"s1","x":712,"y":850,"hoverDwellMs":300}'
→ {"ok":false,"errorCode":"stale_coordinates","error":"stale_coordinates: a raw {x,y} click carries no freshness proof — the page may have scrolled..."}

Adding "atScrollX":0,"atScrollY":0 makes it work. Two problems:

  • The freshness guard exists to stop a stale click landing on the wrong control. A hover is read-only (it moves the pointer to trigger tooltips/snap highlights); refusing it buys no safety, but a coordinate scan of a canvas (hover a grid, read the tooltip after each) now needs the boilerplate on every call.
  • The error text says "click", so when it appears from hover it reads as a mis-routed message. In a scripted loop that only reads the tooltip DOM, the refusal is invisible — the tooltip simply never changes and it looks like the page is "stuck". That is exactly what happened: ~30 minutes of debugging a "sticky tooltip" that was really 300 refused hovers.

Ask: exempt hover from the freshness gate (or default atScrollX/atScrollY to the current scroll for hover), and make the message name the verb it came from. The driving-the-extension skill only documents atScrollX/Y for click; it should say hover too.

2. nbrowser_input_dispatch {"type":"mouseWheel"} silently dispatches a CLICK

adom-bridge --target adom_drew2 nbrowser_input_dispatch '{"sessionId":"s1","type":"mouseWheel","x":56,"y":915,"deltaX":0,"deltaY":-300}'
→ {"ok":true,"dispatched":"click","via":"coords","x":56,"y":915}

Unknown type falls through to a click. On a canvas editor with a tool armed (Add Wires), four "zoom" attempts became four clicks (two of them a double-tap that re-framed the camera). There is currently no verb that scrolls/zooms a canvas: press_key can't (Babylon's ArcRotateCamera zooms on wheel), and page-level scrolling isn't it either. Workaround that works: nbrowser_eval dispatching a synthetic new WheelEvent("wheel",{clientX,clientY,deltaY}) on the <canvas> — Babylon listens with addEventListener so untrusted events are fine.

Ask: reject unknown type values with an error instead of clicking, and add a real wheel (Input.dispatchMouseEvent type:mouseWheel via CDP, isTrusted) — either input_dispatch type:"wheel" or a nbrowser_scroll {x,y,deltaX,deltaY} verb.

3. nbrowser_screenshot ignores toFile; result is base64 only, and pull_file wants filePaths

nbrowser_screenshot '{"sessionId":"s1","toFile":"C:/Users/drew/AppData/Local/Temp/shot.png"}'
→ ok, but no file is written; response carries base64 + width/height
pull_file '{"path":"...shot.png"}'  → "filePaths required (array of absolute Windows paths)"

Minor, but the skill text implies a file path round-trip. Either honor toFile (write on the laptop, return the path) or document that the verb returns base64 and drop toFile from the schema. For the record the base64 path is fine for agents (decode locally); it just should not pretend otherwise.

Environment

  • Session opened with nbrowser_open_window {sessionId,thread,purpose,profile:"chrome:[email protected]"}, driven in the background throughout; every other verb behaved as documented (click with atScrollX/Y, eval, wait_for, close_window).
  • Page: https://hydrogen.adom.inc/drew2/Gallia-test/edit → 3D Editor V2 tab (Babylon canvas 885×1178).
  • Filed by Drew's pipeline-verification session, 2026-09-03.

2 Replies

Drew Owens · 2mo ago

Retested after bringing the laptop to the current builds (nb bridge 0.5.1, extension updated from the stale 0.1.3 to the page's 0.2.9 build, extensionStale:false), so none of this is a version artifact:

  • Item 1 still reproduces on 0.2.9: nbrowser_hover {"sessionId":"s","x":200,"y":300} → stale_coordinates with the "raw {x,y} click" wording.
  • Item 2 still reproduces on 0.2.9: nbrowser_input_dispatch {"type":"mouseWheel",...} → {"ok":true,"dispatched":"click"}.
  • Item 3, correction: on 0.2.9 a file IS written, just not where asked. toFile is ignored and the shot lands in %LOCALAPPDATA%\Temp\adom-nbrowser-shots\shot-<ts>.png, returned as path. So the fix is either honor toFile or document the fixed directory; the base64 is still in the reply either way. (The original report said "no file is written"; that was true of 0.1.3 as loaded, and the response there carried no path.)

Side note for the skill: nbrowser_readiness reports staleOnDisk:[] even when the unpacked folder itself is old, and nbrowser_dev_reload then reloads the same old build. Refreshing the folder from the page's extension/ and reloading fixed it; worth a line in installing-the-extension → Updating.

John Lauer · 2mo ago

All three fixed and verified live on AdomLapper today (bridge 0.16.2 and 0.16.3, extension 0.2.10, both Chrome profiles). Item 1: the stale_coordinates gate now applies to CLICKS only. nbrowser_hover, input_dispatch type:move and the wheel verbs accept raw {x,y} with no atScrollX/Y, because a stale hover shows the wrong tooltip but never presses the wrong control; the refusal text names the verb it came from (a raw {x,y} click ...). Verified: nbrowser_hover {x:300,y:400} with no baseline returned ok:true dispatched:hover; nbrowser_click with the same bare coords still refused with the click wording. Item 2: an unknown input type is REFUSED with errorCode unknown_input_type and dispatched:false, never a click (verified with type:zoom). A real isTrusted wheel exists twice: input_dispatch type:wheel (mousewheel and scroll are accepted aliases, so your original mouseWheel spelling now does what you meant) and the new verb nbrowser_scroll {sessionId, x, y | selector | text, deltaX?, deltaY, steps?}. It dispatches CDP Input.dispatchMouseEvent mouseWheel at the point and reports scrollBefore/scrollAfter/pageScrolled; pageScrolled:false is normal on a canvas that consumed the wheel. Verified on the wiki page: deltaY 300 scrolled 0 to 300, deltaY -300 back to 0. Item 3: toFile:"C:/abs/path.png" is honoured (parent dirs created, path returned); toFile:true keeps the fixed dir under %LOCALAPPDATA%\Temp\adom-nbrowser-shots. Verified: toFile C:/Users/john/AppData/Local/Temp/nb41-proof/shot.png came back as path with savedToFile:true, 250 KB. Side note: readiness staleOnDisk now covers connected profiles too, and installing-the-extension has an Updating section that says which stale signal is which (your 0.1.3 folder showed up under staleProfiles, not staleOnDisk), that dev_reload reloads whatever is in the folder, and that a bare dev_reload only takes the active profile. driving-the-extension documents the clicks-only gate, the wheel verbs and the valid input types. One more thing your session may have hit: nbrowser_close_window timed out at 30s on every call (the window did close) since 0.5.1; fixed in 0.16.3, filed separately. Closing.

John Lauer closed this · 2mo ago

Log in to reply.