← Commit history

skill: document nbrowser_select, shadow-piercing finder, screenshot coord-map, session pinning + eager profile resolve, and the occluded-window focus-emulation click fix

John Lauer ·f946649784 · ·parent 2deb08e
2 files changed +40−1
package.json+1−1
@@ -1,6 +1,6 @@ {   "slug": "adom-browser-extension",-  "version": "0.3.18",+  "version": "0.3.19",   "type": "app",   "title": "Adom Native Browser Extension (Chrome + Edge)",   "description": "Drive your real, logged-in Chrome/Edge from the Adom ecosystem: the nbrowser_* native-browser bridge (counterpart to pup). Multi-profile driving, a read-only credential-library query over the browser's own saved logins (Chrome + Edge), and hands-free vendor login automation, so chip CAD / datasheets download past every login wall in your real signed-in session.",
skills/driving-the-extension/SKILL.md+39
@@ -114,6 +114,25 @@ Prefer these over fixed sleeps + coordinate guessing.   **`changed`** (did the URL/DOM actually change) + `urlBefore`/`urlAfter`. `changed:false` means the   click dispatched but nothing happened (disabled/covered element, missed target) - re-resolve the   selector or retry; don't assume a click worked.+- **Custom elements + shadow DOM just work.** The finder behind `nbrowser_click`/`eval`/`select`+  **pierces open shadow roots** (recursively), so a control whose real options live in a web-component's+  shadow tree (Schwab's account picker, QBO's report search) is findable by **visible text** or a+  **stable attribute** (`[data-testid=…]`) - no more "not found → fall back to fragile+  screenshot-coordinate clicks." (Only *open* shadow roots are reachable; closed ones are invisible to any+  script.)+- **Pick a dropdown option by its TEXT - never coordinate-click a list.** `nbrowser_select+  {label | near, optionText, timeoutMs?}` is the reliable primitive for custom AND native selects: it+  finds the trigger (`label` = the control's accessible name, or `near` = a visible label beside it),+  opens it with an isTrusted click, **waits** for the options to render, clicks the **exact** text match+  (shadow-piercing), and **returns the newly-selected value** (`selected` + `clickedText`) so you can+  confirm it. This is what stops coordinate clicks landing on the wrong row of a virtualized list+  (mis-selecting a bank account by one row). ALWAYS check `selected` before continuing.+- **Background clicks now land on occluded windows (auto).** When the user's own window is maximized+  over our background one, Chrome's occlusion tracking used to make isTrusted CDP clicks silently land+  NOWHERE (eval/screenshots still worked - they don't need a hit-testable frame). The extension now+  emulates focus on attach, so `nbrowser_click`/`type`/`select` land while the window stays in the+  background (no focus theft). If a click ever reports `changed:false` on a page you know is interactive,+  it is almost never occlusion anymore - re-resolve the target or `wait_for` the result. - **Stay signed in.** `nbrowser_keepalive {sessionId, intervalSec?}` replays a benign mousemove/focus on   an interval (default 4 min) to beat vendor idle sign-outs (Gusto/Intuit ~10-15 min). Set it at the   START of a long automation; it auto-stops when the window closes; `{stop:true}` to cancel. (Client@@ -177,6 +196,16 @@ Chrome AND Edge are DISTINCT connections that coexist - the host singleton AND t   browser/profile simultaneously - every verb carries its own explicit target, so there's no shared   default to clobber. `nbrowser_use_profile` only sets the convenience default for verbs that omit a   target; for parallel work, ALWAYS pass an explicit `profile`/`browser`.+- **Pin a session so you can't cross wires.** Driving several accounts and forgetting `profile:` on one+  call is how you act on the wrong profile. `nbrowser_open_window` **auto-pins** its `sessionId` to the+  profile it opened in, so every later verb carrying that sessionId auto-routes back to the SAME profile+  with NO `profile:` needed. Re-pin explicitly with `nbrowser_pin_session {sessionId, profile}`. An explicit+  `profile:` on a call still overrides the pin.+- **Resolve emails up front.** A freshly-connected profile can briefly key as `pending-N` with an empty+  email until its Google identity resolves (the extension now retries + re-announces automatically). To+  force it before you drive, call `nbrowser_resolve_profiles` (all connected profiles) or+  `nbrowser_whoami {profile}` (one) - keys become `chrome:<email>` / `edge:<email>`. A profile with no email+  isn't signed into Google; target it by its id-key or by `browser:`. - **Exclude one:** `nbrowser_profile_block {profile}` (persisted) - a security whitelist that keeps a   browser/profile from ever being driven. @@ -234,6 +263,16 @@ adom-desktop -t AdomLapper nbrowser_screenshot '{"sessionId":"s1","toFile":true} Screenshots are large base64 - over the relay they can time out, so **prefer `toFile:true` + `pull_file`** (binary, chunked). `screenshot_full_res` is unresized; default resizes ≤1568 for the model. +Every screenshot response carries a **`viewport`** coordinate-map sampled at capture time, so you can turn+a screenshot pixel into a click coordinate EXACTLY without a separate `window.innerWidth`/`dpr` probe (and+detect scroll drift): `{innerWidth, innerHeight, devicePixelRatio, scrollX, scrollY, capturedWidth,+capturedHeight, imageWidth, imageHeight, resizeRatio, cssPerImageX, cssPerImageY, fullPage}`. To click a+feature you see at pixel `(px,py)` in the returned image: **`clickX = px * cssPerImageX`,+`clickY = py * cssPerImageY`** (viewport CSS px, what `nbrowser_click {x,y}` takes). Before clicking, re-read+`scrollX/scrollY` (or just re-screenshot) - if the page scrolled since the shot, a viewport pixel now points+at different content (that's the drift you were hitting). For a `fullPage:true` shot the map spans the whole+page, so those coords are page-absolute.+ > **🚫 NEVER foreground the window to take a screenshot.** `nbrowser_screenshot` (CDP) captures the > page bitmap with **zero OS-window interaction**, and AD `desktop_screenshot_window` (WGC) captures a > **background / occluded** window fine. The user is usually working *in* their real browser - yanking