app

Adom Browser Extension

Public Made by Adom

by adom

Your AI's hands in your real, signed-in browser. The Adom Browser Extension (abe) works with both Chrome and Edge: build, drive, and test web apps, and log past vendor walls, in the browser you already use - cookies, SSO, saved logins and captcha trust intact. The nbrowser_* bridge (pup's counterpart) runs in your actual logged-in profile.

browserextensionchromeedgenbrowserai-agentdeveloper-toolsvibe-codingnative-messagingadom-desktop
Overview Files Skills Discover Packages Releases Dependencies Issues1 Breadcrumbs Pull Requests Activity
adom-browser-extension / skills / driving-the-extension / SKILL.md
Download

name: driving-the-extension description: How to drive the Adom browser extension (nbrowser_* verbs) - control the user's REAL logged-in Chrome/Edge from the cloud. Covers the CDP-vs-native split (and the "debugging this browser" bar), the full verb surface, isTrusted input (click-by-text + xy-finding, type/press_key, hover, right-click), the gliding Adom cursor with intent narration, screenshots, the taskbar status feature (flash/progress/Adom-overlay), background-safe capture (never stealing the user's focus), captions, the chip-fetcher use case, and every pitfall learned building it.

Driving the Adom Browser Extension (nbrowser_*)

This extension turns the user's real, logged-in Chrome/Edge into something the agent drives - the native-browser counterpart to pup. Because it runs inside the user's actual profile, their cookies, SSO, saved logins and human-trust signals all apply, so it sails through login walls and captchas that block pup's empty controlled-Chromium. That is the whole point.

⛔ Golden rule: NEVER reuse a window the user is in - always open a fresh one

The user is working in their browser. Never navigate, reload, or repurpose a window/tab they already have open - not with nbrowser_navigate, and not with the OS-level desktop_navigate either. That yanks their live page out from under them and is rightly seen as hostile. This applies to everything you do for your own needs - driving a workflow, reading a doc, and especially grabbing a screenshot (e.g. of a chrome://extensions / edge://extensions page for a guide).

The rule, every time: open a NEW window (nbrowser_open_window {sessionId, thread, purpose, url}, or a fresh OS window you then desktop_navigate), do your work and capture in THAT window, and close it when done (nbrowser_close_window {sessionId, includeSpawned:true}). The only thing you may do to one of their existing windows is desktop_bring_to_front to surface it for them - never to drive or navigate it.

As of extension v0.0.21 this rule is ENFORCED IN CODE, not just documented. Every mutating verb (navigate, reload, back, forward, click, type, press_key, hover, input_dispatch, select, set_date, eval, cursor, caption, focus, close_tab, switch_tab, wait_for, totp, autologin, login_form, record_start) refuses:

  • a bare call (no sessionId/tabId) → no_target — the old active-tab fallback is gone; it is how an agent navigated a user's working tab and wiped their other AI threads;
  • a target outside a window the agent opened → not_owned (nbrowser_open_window's session map IS the ownership registry). open_tab likewise requires an owned session's sessionId. There is NO agent-passable override. Refusals carry a teaching _hint with the corrected flow, and the first response of each connection carries a one-time _safety banner stating this rule. (tools/check-guard-coverage.mjs fails the build if a new verb ships unclassified or ungated.)

⭐ MANDATORY PREFLIGHT — profiles first, then your own window

Before driving ANYTHING, in this order:

  1. nbrowser_profiles (read-only) → the connected browser:email keys (Chrome AND Edge). Each row also carries displayName — the human label the USER sees on the browser's own profile button ("John Personal", "adom.inc") — plus profileDir and hasAvatar (v0.1.20+). Never assume which browser holds an account — the currently-active browser is NOT "the" browser. A row with unresolved:true hasn't announced its email yet (fresh bridge restart): run nbrowser_resolve_profiles FIRST or you may drive the wrong account. A row with asleep:true EXISTS on the machine but has no open window (its extension service worker sleeps), so it can't be driven yet — wake it yourself with nbrowser_wake_profile {profile:"<key>"}: it launches the browser with --no-startup-window, so the profile + extension load in the BACKGROUND — no window opens, nothing foregrounds, no focus is stolen — and the profile flips to live in ~2s. NEVER ask the human to open a window for you, and NEVER launch a foreground browser to wake a profile — the wake verb exists precisely so waking honors background-by-default. (woke:false usually means the Adom extension isn't installed in THAT profile — extensions are per-profile; offer the install.)
  2. Pick the profile whose email matches the identity the task needs — email DOMAIN + displayName tell you work vs personal (corporate domain = work; gmail.com etc. = personal). Setting up a tool for [email protected] → drive chrome:[email protected], not whatever profile happens to be focused. If the needed profile isn't listed, ask the user to open a window in it — don't improvise in another one.
  3. nbrowser_open_window {sessionId:"<task id>", thread:"<which AI thread>", purpose:"<why, human-readable>", profile:"<key>", url:…} → a BACKGROUND window auto-pinned to that profile. All three of sessionId/thread/purpose are REQUIRED (registration) or it refuses session_unregistered; purpose shows in the Session Monitor.
  4. Drive ONLY via that sessionId. If a page you drive spawns a window (a Gmail-Contacts email click pops a compose; target=_blank/OAuth popups), your next verb is refused unacknowledged_spawns → nbrowser_owned {sessionId} to see it, nbrowser_cleanup {sessionId} to close the litter.
  5. End the task by closing what you own: nbrowser_close_window {sessionId, includeSpawned:true}.

⛔ Background by default - NEVER foreground unless the user asks

The user is almost always doing something else on their computer - reading, typing, on a call, in another app. So your work MUST default to the background, especially when you open new browser windows: a window that jumps to the front interrupts whatever they were doing and (worse) their next keystrokes land in the control you just surfaced. Background is not politeness, it is the default posture; foreground is opt-in, only when the user explicitly asks to watch. Never raise a window to the foreground for your own needs. Foregrounding STEALS the user's keyboard focus - if they are typing, your window captures their keystrokes (it has typed a user's message into a username field and killed Chrome's password dropdown). As of v0.1.0 a bare userRequestedForeground:true is not enough: without a foregroundReason the window stays background. To genuinely foreground (some auth pages need it): first adom-desktop notify_user to warn the user, then pass userRequestedForeground:true + foregroundReason:"<why>", do the one thing, and return to background. The verbs are built for this: nbrowser_open_window opens unfocused, open_tab opens active:false, switch_window does not raise - leave those defaults alone. Capture without foregrounding: nbrowser_screenshot (CDP) and desktop_screenshot_window (WGC) both capture a background/occluded window, and WGC records one at full fps (with the occlusion flags). So there is never a reason to pull a window forward to grab a shot or a recording.

Capturing a chrome:// / edge:// page (e.g. chrome://extensions for a guide) is the classic trap: do NOT launch the browser yourself - msedge / chrome --new-window (via launch / launch_process) raises the new window to the foreground. Instead:

  1. nbrowser_open_window {sessionId:"guide", thread:"<you>", purpose:"screenshot chrome://extensions for a guide", url:"about:blank"} → a fresh background window (note its hwnd).
  2. desktop_navigate {hwnd, url:"chrome://extensions"} → loads the chrome:// page in the background, no foreground.
  3. desktop_screenshot_window {hwnd} → WGC captures it where it sits.
  4. nbrowser_close_window {sessionId}.

Foregrounding is the user's choice - only desktop_bring_to_front, and only when they explicitly ask to watch live.

⚠️ Foreground can happen even when you NEVER call bring_to_front. Opening a new window, and especially desktop_navigate (its omnibox commit can surface/activate the window), may bring a window to the FRONT despite focused:false. Judge by the outcome, not the verb: after an open or a navigate, check desktop_list_windows (a window at z:0 is topmost = you surfaced it) and treat that as a disruption. Observed live: a desktop_navigate to chrome://extensions left the window at z:0 with no bring_to_front called.

Path: cloud → relay → adom-desktop (laptop) → native-browser bridge → native-messaging host → extension service worker → back. You call adom-desktop nbrowser_* '<json>'.

Targeting (READ FIRST)

If more than one Adom Desktop is on the relay (laptop AdomLapper + the winvm), every call must name one or it errors ambiguous_target:

adom-desktop --target AdomLapper nbrowser_status

A "session" = a real Chrome window (sessionId you choose, or auto s1…). tabId = the real chrome tab id. The session→window map persists to chrome.storage, so sessions survive reloads.

⭐ The #1 thing to understand: CDP vs native (and the "debugging this browser" bar)

The extension is a CDP-primary hybrid with two engines:

  • chrome.debugger (CDP) - needed for real isTrusted input, full-page screenshots, and page eval. Attaching it is what makes Chrome show the yellow "Adom for Chrome & Edge" started debugging this browser bar. The bridge lazy-attaches only for CDP verbs and idle-detaches ~8s later, so the bar appears during active clicking/typing and clears when idle.
  • Native chrome.* APIs - windows/tabs/navigation/downloads. No debugger → no bar.
🟡 Shows the bar (CDP) ⚪ No bar (native)
click, double_click, right_click, hover open_window, close_window, switch_window, list_windows
type, press_key, input_dispatch, cursor open_tab, close_tab, switch_tab, list_tabs, rescan
screenshot, screenshot_full_res navigate, reload, back, forward, wait
eval / evaluate, errors fetch_url, download, caption, focus, flash/taskbar

The bar is a Chrome security guarantee and cannot be hidden on consumer Chrome while driving. On Adom-managed machines the --silent-debugger-extension-api launch flag removes it. If you only need to read/navigate, prefer native verbs and the bar never shows.

Verb surface

  • meta: status, ping, help (full inline reference), dev_reload (self-completing 2-pass; throttled 1/90s)
  • windows/sessions: open_window (REQUIRES sessionId+thread+purpose), close_window (includeSpawned:true also closes spawns), switch_window, list_windows (returns bounds+state+an owner label agent-opened|agent-spawned|user; mine:true filters to yours), window_state/fullscreen (maximize·fullscreen·normal·minimize, or left/top/width/height to move+resize - native, no bar; F11-via-key does NOT work), rescan, resync_sessions
  • ownership / cleanup: owned {sessionId} (opened + SPAWNED windows/tabs a driven page opened; acknowledges spawns) · cleanup {sessionId} (close ONLY what this session spawned, never the user's) · force_close {windowId} (two-step attested last resort for a TRUE orphan; never another thread's window)
  • account / credentials: login_state {url} (signed-in NOW? via chrome.cookies) · credentials {match?, browser?, profile?} - a read-only QUERY of the browser's OWN saved-login store (we don't store or replace anything; Chrome/Edge stay the source of truth). Returns {browser, profile, host, username} rows across Chrome AND Edge, all profiles - i.e. which vendors the user can auto-login to. Never passwords (the browser autofills those). The extension can't read this (sandbox); the bridge reads each browser's Login Data SQLite natively.
  • reliable automation: wait_for, keepalive, robust click (returns changed), set_date, download_wait, totp/register_totp (2FA), add_extension_origin - see Reliable automation below.

Reliable automation (waits, click-verify, 2FA, downloads, dates)

The primitives that make an unattended multi-vendor pull (Gusto/QuickBooks/Carta) actually hold up. Prefer these over fixed sleeps + coordinate guessing.

  • Deterministic waits, not sleeps. After a click/nav, nbrowser_wait_for {sessionId, selector | urlIncludes | textIncludes, timeoutMs?} blocks until the thing you're waiting for exists, then returns {matched, by, url, waitedMs}. matched:false, timedOut:true = it never appeared - screenshot to see why. Never race a fixed nbrowser_wait {ms} against a page load.
  • Click, then verify it landed. nbrowser_click now scrolls the target into view and returns changed (did the URL/DOM actually change) + urlBefore/urlAfter. changed:false means the click dispatched but nothing happened (disabled/covered element, missed target) - re-resolve the selector or retry; don't assume a click worked.
  • Custom elements + shadow DOM just work. The finder behind nbrowser_click/eval/select pierces open shadow roots (recursively), so a control whose real options live in a web-component's shadow tree (Schwab's account picker, QBO's report search) is findable by visible text or a stable attribute ([data-testid=…]) - no more "not found → fall back to fragile screenshot-coordinate clicks." (Only open shadow roots are reachable; closed ones are invisible to any script.)
  • Pick a dropdown option by its TEXT - never coordinate-click a list. nbrowser_select {label | near, optionText, timeoutMs?} is the reliable primitive for custom AND native selects: it finds the trigger (label = the control's accessible name, or near = a visible label beside it), opens it with an isTrusted click, waits for the options to render, clicks the exact text match (shadow-piercing), and returns the newly-selected value (selected + clickedText) so you can confirm it. This is what stops coordinate clicks landing on the wrong row of a virtualized list (mis-selecting a bank account by one row). ALWAYS check selected before continuing.
  • Background clicks now land on occluded windows (auto). When the user's own window is maximized over our background one, Chrome's occlusion tracking used to make isTrusted CDP clicks silently land NOWHERE (eval/screenshots still worked - they don't need a hit-testable frame). The extension now emulates focus on attach, so nbrowser_click/type/select land while the window stays in the background (no focus theft). If a click ever reports changed:false on a page you know is interactive, it is almost never occlusion anymore - re-resolve the target or wait_for the result.
  • Stay signed in. nbrowser_keepalive {sessionId, intervalSec?} replays a benign mousemove/focus on an interval (default 4 min) to beat vendor idle sign-outs (Gusto/Intuit ~10-15 min). Set it at the START of a long automation; it auto-stops when the window closes; {stop:true} to cancel. (Client activity only - a vendor that expires the SERVER session regardless may still need periodic real actions.)
  • 2FA / TOTP - the unattended-automation unblock. One-time: nbrowser_register_totp {host, seed} with the vendor's base32 "manual entry / setup key" (shown by the 2FA QR). It's stored 0600 on the laptop and NEVER returned. Then, when a 2FA prompt appears, focus the code field (or pass selector) and nbrowser_totp {host, submit?} - the bridge computes the RFC-6238 code and the extension types it; the agent never sees the seed or the code. errorCode:"no_totp_seed" = register it first. Codes rotate every 30 s, so call it right when you need it.
  • Downloads by event, not by polling. After you trigger an export, nbrowser_download_wait {sessionId, timeoutMs?, sinceMs?} returns the landed files[] via chrome.downloads events (no timestamp-guessing the Downloads folder). Pass saveToSubfolder:"<name>" to route subsequent downloads into a subfolder of Downloads.
  • Segmented date fields. Finance apps use react-aria spinbutton date fields (month/day/year). nbrowser_set_date {selector, date:"YYYY-MM-DD", order?} focuses the field and types each digit as a real keydown so the segments advance. US MDY by default; pass order:"DMY" for day-first. Check fieldValue in the response.
  • Edge different-ID lockout. If the extension is loaded in Edge but never shows in nbrowser_profiles, its loaded-unpacked ID may differ from the one in the native-host manifest's allowed_origins. Copy the ID off edge://extensions and nbrowser_add_extension_origin {id}, then reload the extension so the browser re-reads it. (Our manifest pins a key, so IDs should match - use this only if Edge diverged.)

Logging into vendors (the key to fetching electronics data)

Every EE vendor (DigiKey, Mouser, SnapMagic, UltraLibrarian, ComponentSearchEngine, TI, NXP, Arrow…) gates CAD/datasheet downloads behind a login. You can automate all of it without ever seeing a password:

  1. Know what you can log into: nbrowser_credentials {match:"digikey"} → if the vendor is in the library, the user has a saved login and Chrome will autofill it. (No match → ask the user to save it once, or pick another rung of the sourcing ladder.)
  2. Drive the login: nbrowser_navigate to the sign-in page → nbrowser_click the email/username field (Chrome autofills username+password - you never see the secret) → nbrowser_click "Sign In"/"Log In" → nbrowser_login_state to confirm. Proven live on Mouser (johnAdom) + CSE.
  3. Signed in = no bot wall: a real logged-in session also skips the anti-bot "security slider" captchas that block pup's empty profile. That's the whole point of driving the real browser. Mouser uses a Username field (not email) - check input by name, not just type=email.
  • tabs: open_tab, close_tab, switch_tab, list_tabs
  • navigation: navigate, reload, back, forward, wait{ms}
  • input (isTrusted, CDP): click, double_click, right_click, hover, type, press_key, input_dispatch, cursor
  • capture/eval: screenshot, screenshot_full_res, eval/evaluate
  • data (native advantage): fetch_url (uses the user's REAL cookies - authed), download
  • status/errors: errors, events (window/tab closes incl. byUser:true)
  • demo/UX: caption, focus, flash/flash_stop/flash_mode/taskbar
  • recording: record_start/stop/status/list - ⭐ use method:"wgc" (delegates to the core AD verb desktop_record_window_start: background, 60fps, no picker, no banner, captures occluded). getdisplaymedia/tabcapture are a refused-by-default fallback (Chrome forces a manual share picker) - need fallbackConfirmed:true. See native-browser-recording.

Parallel multi-browser / multi-profile control

Every BROWSER × PROFILE is its own independently-addressable connection, keyed browser:email (e.g. chrome:[email protected], edge:[email protected], chrome:[email protected]). The same account signed into Chrome AND Edge are DISTINCT connections that coexist - the host singleton AND the bridge both key by browser:profile, so the 2nd browser's host no longer exits with "Native host has exited."

  • List them: nbrowser_profiles → each entry's profile field is its addressable key, plus browser, email, live, active.
  • Target any of them, in PARALLEL: pass profile:"<key>" on ANY verb - nbrowser_navigate {profile:"edge:[email protected]", url:…}. Also accepts profile:"<email>"+ browser:"edge", or browser:"edge" alone (that browser's active profile). A wrong/absent target no-routes cleanly (doesn't fall through to the wrong browser).
  • Multiple AI threads at once: separate Claude/Codex threads can each drive their own browser/profile simultaneously - every verb carries its own explicit target, so there's no shared default to clobber. nbrowser_use_profile only sets the convenience default for verbs that omit a target; for parallel work, ALWAYS pass an explicit profile/browser.
  • Pin a session so you can't cross wires. Driving several accounts and forgetting profile: on one call is how you act on the wrong profile. nbrowser_open_window auto-pins its sessionId to the profile it opened in, so every later verb carrying that sessionId auto-routes back to the SAME profile with NO profile: needed. Re-pin explicitly with nbrowser_pin_session {sessionId, profile}. An explicit profile: on a call still overrides the pin.
  • Resolve emails up front. A freshly-connected profile can briefly key as pending-N with an empty email until its Google identity resolves (the extension now retries + re-announces automatically). To force it before you drive, call nbrowser_resolve_profiles (all connected profiles) or nbrowser_whoami {profile} (one) - keys become chrome:<email> / edge:<email>. A profile with no email isn't signed into Google; target it by its id-key or by browser:.
  • Exclude one: nbrowser_profile_block {profile} (persisted) - a security whitelist that keeps a browser/profile from ever being driven.

For a browser to appear, that profile needs an open window AND the Adom extension loaded in THAT browser (chrome://extensions / edge://extensions → Developer mode → Load unpacked → the extension folder). The hello carries the browser (Chrome vs Edge via the UA), so each connection self-keys.

Input - click, type, hover, right-click, and how xy is found

You don't guess pixels. The smart finder scores the DOM, takes the matched element's getBoundingClientRect, and computes its center xy - then CDP dispatches the isTrusted event there. Targeting is DOM-truth, not pixel-guessing.

Why DPI / window-resize never breaks targeting (the question every AI asks)

This is not screenshot→eyeball-a-pixel→click automation, so it does not suffer that class of bug:

  • High-DPI / display scaling is handled by Chrome, not you. getBoundingClientRect and CDP Input.dispatchMouseEvent are BOTH in CSS pixels, viewport-relative - the same space. The browser maps CSS→physical internally. A 4K@200% screen and a 1080p screen give the identical xy for the same button. There is zero DPI-translation code, hence zero DPI bugs. Never touch devicePixelRatio.
  • Screen resolution / window size / window position are irrelevant - targeting is page-relative, not screen-relative. Drag Chrome to the other monitor mid-task; it doesn't matter.
  • No stale-coordinate / "dirty flag" problem. Measure and act are one atomic verb, sub-second, with no AI loop between them - the finder re-measures getBoundingClientRect fresh on every click/ type/hover. A user resizing the window 100ms before your click is fine: the next call reads the live rect. Nothing is cached across a gap. So you can reassure the user that resizing/scrolling while Adom drives is SAFE (a nice nbrowser_caption) - the opposite of the usual "don't touch anything" fear.
  • The only path that CAN go stale is passing raw {x,y} to input_dispatch (a literal pixel, no re-resolve). Prefer text/selector - they re-resolve every call. AD is not involved in click xy at all (it only does OS-level foreground/taskbar/window-screenshots). This is now ENFORCED (stale_coordinates, v0.1.0): a raw {x,y} click is refused unless you pass atScrollX/atScrollY from your latest screenshot's coordMap AND the page hasn't scrolled since (the "Next" pixel that became "Back" after a re-render). Just click by text/selector.
  • Click-by-text that matches >1 visible element is refused (ambiguous_target, v0.1.0) - two "Next"/"Back" buttons, three "Yes" radios. Pass a unique selector, or {debug:true} to see the numbered candidate overlay and pick one, or acceptAmbiguous:true to take the top-scored match. Verify matchedText/changed in the response after a text click.
# click by VISIBLE TEXT (finder computes xy) - returns {x,y,matchedText}
adom-desktop -t AdomLapper nbrowser_click '{"sessionId":"s1","text":"Pay","cursor":true,"intent":"Clicking the Pay button"}'
adom-desktop -t AdomLapper nbrowser_click '{"sessionId":"s1","selector":"#download"}'         # or by selector
adom-desktop -t AdomLapper nbrowser_type  '{"sessionId":"s1","selector":"#search","text":"STM32F103C8T6"}'
adom-desktop -t AdomLapper nbrowser_press_key '{"sessionId":"s1","key":"Enter"}'              # Enter/Tab/ArrowDown/…
adom-desktop -t AdomLapper nbrowser_hover '{"sessionId":"s1","text":"Hover me"}'              # fires :hover + tooltips
adom-desktop -t AdomLapper nbrowser_hover '{"sessionId":"s1","selector":".info-icon"}'        # or by selector (surest for hover)
adom-desktop -t AdomLapper nbrowser_right_click '{"sessionId":"s1","selector":".ctx"}'        # real context menu

Hover targets are often NON-interactive (info icons, cards, tooltip triggers, plain spans). The finder first scans clickable elements (button/a/input/…); if text matches none, it falls back to the smallest visible element whose text matches, so hover {text:"…"} works on a bare <span>. A CSS :hover/JS mouseenter tooltip needs the pointer to enter and dwell, so hover does an approach-move → settle → ~600ms dwell (tune with hoverDwellMs). When in doubt for hover, a selector is surest.

The gliding Adom cursor: pass cursor:true + intent:"…" on any input verb and the gorgeous Adom cursor (teal arrow + pulsing ring + "Adom" label) glides to the target while narrating its intention ("Clicking the Pay button", "Hovering the card to reveal the tooltip"). Builds user trust - they always see what the AI is doing and why. Mimics Hydrogen Desktop's cursor.

Native JS dialogs (alert / confirm / prompt) - the invisible blocker

A native alert()/confirm()/prompt()/beforeunload is not an OS window (no hwnd) and does not appear in a screenshot (Chrome renders it as browser chrome), but it FREEZES the page's main thread. Symptom: your clicks/typing/eval suddenly do nothing and you "can't fill the form" (real incident: an Intuit "questionnaire saved" alert sat over the form while the AI kept trying). ABE now auto-detects it: the blocked verb returns modal_open (with the dialog text) and a screenshot carries a _modal note. Dismiss it: nbrowser_handle_dialog {sessionId, accept:true} (OK/Yes/Leave), accept:false (Cancel/Stay), or text:"…" to answer a prompt - then re-screenshot and continue. If an action mysteriously stops working, suspect a dialog.

Screenshots — pick the RIGHT one (page vs window vs NEVER fullscreen)

Three ways to "see", and the choice matters:

  • The PAGE → nbrowser_screenshot {sessionId} (CDP). Your DEFAULT. Fast, background-safe, captures the page bitmap. But it shows page content ONLY - no browser chrome (URL/tabs) and no native popups/ dialogs (Chrome's saved-password dropdown, print dialog, alert/confirm) which render OUTSIDE the page.
  • The WINDOW + chrome + native popups → desktop_screenshot_window {hwnd | titleContains} (AD, WGC). Use this when you need the URL bar, a chrome:// page, or a native popup/dialog. It captures the window plus its OWNED CHILD windows and returns the full parent/child hwnd array - the same technique that makes KiCad/Fusion error dialogs visible. Background-safe (no foregrounding). nbrowser_open_window returns the window's hwnd for exactly this; or find it via desktop_list_windows by title.
  • ⛔ NEVER a fullscreen / whole-monitor screenshot to inspect a browser window. It grabs the user's other work, is imprecise, and usually implies foregrounding. There is always a targeted option above. (nbrowser_screenshot_full_res is full-PAGE, not full-monitor - it still only sees the page.)
adom-desktop -t AdomLapper nbrowser_screenshot '{"sessionId":"s1","toFile":true}'   # PAGE bitmap → laptop PNG path
adom-desktop -t AdomLapper desktop_screenshot_window '{"hwnd":<from open_window>}'   # WINDOW + chrome + child popups
# then: adom-desktop -t AdomLapper pull_file '{"filePaths":["<path>"],"saveTo":"/tmp"}'

Screenshots are large base64 - over the relay they can time out, so prefer toFile:true + pull_file (binary, chunked). screenshot_full_res is unresized; default resizes ≤1568 for the model.

Every screenshot response carries a viewport coordinate-map sampled at capture time, so you can turn a screenshot pixel into a click coordinate EXACTLY without a separate window.innerWidth/dpr probe (and detect scroll drift): {innerWidth, innerHeight, devicePixelRatio, scrollX, scrollY, capturedWidth, capturedHeight, imageWidth, imageHeight, resizeRatio, cssPerImageX, cssPerImageY, fullPage}. To click a feature you see at pixel (px,py) in the returned image: clickX = px * cssPerImageX, clickY = py * cssPerImageY (viewport CSS px, what nbrowser_click {x,y} takes). Before clicking, re-read scrollX/scrollY (or just re-screenshot) - if the page scrolled since the shot, a viewport pixel now points at different content (that's the drift you were hitting). For a fullPage:true shot the map spans the whole page, so those coords are page-absolute.

🚫 NEVER foreground the window to take a screenshot. nbrowser_screenshot (CDP) captures the page bitmap with zero OS-window interaction, and AD desktop_screenshot_window (WGC) captures a background / occluded window fine. The user is usually working in their real browser - yanking it forward to grab a shot disrupts them. See Do NOT foreground the window.

Taskbar status - flash, progress, the "Adom is driving" overlay badge

Paint glanceable status on the real browser window's Windows taskbar button (the hwnd is captured at open_window; an Adom overlay badge goes on automatically while driving - a trust tell). It lands on opened windows at open, and on adopted SPAWNED windows when you call nbrowser_owned / nbrowser_list_windows (that is when the bridge learns their hwnd). Driven via AD's desktop_taskbar.

What the badge actually is (v0.1.12+): Chrome/Edge use the taskbar overlay slot for the profile avatar, so a bare Adom badge would evict it and the user loses "which profile is this?". The bridge therefore builds a composite: the profile's own avatar circle-cropped (exactly how the browser draws it) with the Adom favicon in its lower-right corner - one glance says both whose profile and Adom is driving it. It works for Chrome AND Edge (each browser's own avatar file), is composited natively on the user's machine, and is cached per profile.

Badge lifecycle - you do NOT manage it:

  • Applied automatically at open_window and re-asserted by drive verbs - on EVERY addressing path: sessionId-keyed AND profile-keyed (profile: with no sessionId) both count as driving. It also survives bridge restarts (session-window mappings are persisted and restored).
  • Idle-expires after ~3 min of no commands: the button repaints the plain native avatar (the overlay slot is NEVER blanked - blanking would erase the user's own avatar).
  • overlay:{badge:"none"} = "restore native look" (safe); overlay:{badge:"adom"} = re-apply the composite (only needed if a badge went missing, e.g. immersive fullscreen hid the taskbar).
  • On close_window / bridge restart, stale badges are swept by repainting native avatars - never by clearing slots.
  • nbrowser_flash {sessionId} / nbrowser_flash_stop - orange "come look" cue (auto-clears on focus).
  • nbrowser_flash_mode {mode} - quiet (default; auto-flash on come-look moments: page load, screenshot, an error/refusal, a human-gate, and freshly SPAWNED windows - flash MANUALLY with nbrowser_flash at task-done) · chatty (flash every action, for users who want to watch) · off (NO auto-flash at all - the user opt-out; set this the moment the user says the flashing is annoying).
  • nbrowser_taskbar {progress:{state,value?}, overlay:{badge,tooltip?}} - progress states: normal(green) · paused(YELLOW - use for captcha/2FA/login human-gates) · error(red) · indeterminate(marquee) · none. Overlay badge:"adom"|"none" (see lifecycle above).

No extension in the profile? You can still OPEN their real browser (nbrowser_open_os_window)

When a task only needs the user's real browser OPENED at a URL in the right profile - above all OAUTH CONSENT (adom-google, Autodesk APS: the user is already signed in; they just click Approve and your CLI's token poll picks up the result) - you do NOT need the extension installed: nbrowser_open_os_window {sessionId, thread, purpose, profile, url} (registration required, like every window). It opens BACKGROUND by default (v0.1.41+): it launches the window the same way Hydrogen Desktop's browser picker does (a clean launch, no --profile-directory=Default), lets it fully build, then minimizes it once with a gentle no-activate call - so it never steals focus and, importantly, never crashes the browser (an earlier version minimized mid-construction in a tight loop and crashed the whole Edge process - fixed). Its taskbar button flashes ORANGE so the user clicks it when THEY are ready. To raise it later: desktop_bring_to_front {hwnd} (returned in the result) - only when the user asks to see it. foreground:true needs foregroundReason + a notify_user first, same as everywhere else.

  • You CANNOT read, drive, or screenshot that window (no extension there) - poll the outcome out-of-band or ask the user.
  • Close it when the flow is done: nbrowser_close_window {sessionId} (handled bridge-side for these).
  • Every response carries the honest limitation + the ABE upsell - relay it to the user at a natural moment ("with the extension installed I could have filled this for you"), never mid-auth.
  • If the profile HAS the extension, prefer nbrowser_wake_profile + nbrowser_open_window - full control, guards, and cleanup beat a fire-and-forget open every time.

File uploads (nbrowser_set_file_input) - never the OS picker

Upload into an <input type=file> over CDP - background, silent, no OS file dialog (which cannot be driven with the debugger attached and would foreground). Three shapes:

  • Direct (the default): {sessionId, selector|text, filePaths:["C:/abs/laptop/path"]} - the finder walks a Browse button/label to its associated input; HIDDEN inputs work (most real UIs hide them). input + change are dispatched for you so React/Angular notice.
  • Transient inputs (created inside the Browse click handler - nothing exists to select): {armForNextChooser:true, filePaths} then nbrowser_click the Browse control - the page's file chooser is satisfied silently (30s auto-disarm). Confirm with {chooserStatus:true} or a screenshot.
  • Cloud-direct small files: files:[{name, contentBase64}] - the bridge stages them in a janitored temp dir on the laptop and substitutes the paths. One call, no send_files round-trip. Use adom-desktop send_files + filePaths for big payloads. Errors teach: file_missing (bridge stats every path first), input_not_found (with a page diagnosis), no_files. Hard limitation: sites using the File System Access API (showOpenFilePicker) cannot be driven over CDP at all - use the site's drag-drop zone manually. After injection, most UIs still need their Submit/Upload button clicked.

OS-level window inventory (nbrowser_os_windows) + extension-state diagnosis

nbrowser_os_windows {} returns every Chrome/Edge/pup window as the LAPTOP sees it (hwnd, title, rect, z), annotated with abeSession (the live ABE session that owns it - NEVER force-close those) and kind (pup/chrome-for-testing = browser_open/pup litter). It is measured bridge-side, so it works when your own desktop_list_windows returns 0/null - usually AD mid-self-update-restart, or a wrong --target when several ADs are connected. ok:false means "AD link down, data UNKNOWN", never "no windows". Use it to find litter for the nbrowser_force_close {hwnd} gate, and to disambiguate "separate windows vs tabs" without asking the user.

No more not-connected mysteries (v0.1.26+): when a verb is refused extension not connected for a profile that exists locally, the refusal carries a DIAGNOSIS from the browser's own records: the Adom extension is not installed in that profile (fix: installing-the-extension, Load unpacked into THAT profile) vs installed but disabled (fix: enable the toggle) vs enabled but not dialing (fix: retry nbrowser_wake_profile; if it never connects, nbrowser_add_extension_origin {id} for that browser's extension id). nbrowser_wake_profile's woke:false includes the same extensionState, and nbrowser_profiles asleep rows carry extensionInstalled / extensionEnabled / fix - read them BEFORE trying to wake, and never report "the profile doesn't exist" or "can't be fixed" when these fields tell you exactly what to do.

Activity log - "who drove what, when" (nbrowser_activity)

The bridge keeps a bounded in-memory trail (max 600 entries / ~1h TTL; resets on bridge restart) of every forwarded verb: {agoMs, verb, sessionId, thread, profile, ok, errorCode} plus an activeSessions summary (each with lastAgoMs). Use it to answer "why is that profile badged?", "what thread is driving right now?", or to debug a stray window. Filter with {sessionId, thread, profile, verb, errorsOnly, sinceMs, limit}. A session with a large lastAgoMs is idle - its badge has already idle-expired.

Play it like a status light: indeterminate green while working → yellow + flash on a human-gate → clear

  • one flash on done. Respect the user's mode (anxiety-driven users want chatty; trusting users want quiet). Never leave it strobing.

Do NOT foreground the window to look at it - capture in place

The user is usually working IN their real browser. Stealing its focus disrupts them - this has been called out directly ("that's disrupting my work massively"). The whole point of native-browser driving is that the user keeps working in their foreground window while Adom operates one of their other windows in the background. So:

  • Screenshots NEVER need foregrounding - nbrowser_screenshot (CDP) captures a background / occluded window unconditionally (it reads the page bitmap, not the OS window). Capture where it sits; never raise it.
  • ⭐ Recording in the BACKGROUND is the whole point - and it works, with one setup requirement. The WGC recorder (and desktop_screenshot_window's WGC path) capture the OS window's composited surface. Chrome, by default, stops painting a fully-occluded window (its occlusion detector), so WGC then records a blank surface. The fix is a one-time launch-flag requirement on the Chrome being recorded: --disable-backgrounding-occluded-windows --disable-renderer-backgrounding --disable-background-timer-throttling --disable-features=CalculateNativeWinOcclusion. pup's Chrome already ships these (which is why pup background-records reliably); a default user Chrome does not - so for the extension to record the user's real Chrome in the background, that Chrome must be launched with these flags (bake them into the Adom-managed Chrome launch / golden image). Never work around a blank background recording by foregrounding the window - that defeats the entire feature (the user must be able to keep working while a demo records). Foregrounding is the user's choice, when they want to watch - never the AI's.
  • open_window opens in the BACKGROUND (focus is not stolen). Pass focused:true only when the user explicitly asked to watch it live.
  • nbrowser_focus activates the right tab and flashes its taskbar button, but by design does NOT raise the window above other apps (Windows foreground-lock) - so it never yanks the user's focus. To say "come look" without forcing the window forward, use the taskbar flash / overlay above.
  • Only raise a window when the user EXPLICITLY asks to see it live ("show me", "let me watch"):
adom-desktop -t AdomLapper desktop_bring_to_front '{"titleContains":"<page title>","state":"restore"}'  # or {hwnd}

🎥 Recording doctrine - full-framerate background recording of the NATIVE browser

Goal: demos must look amazing → full-resolution, high framerate (20-60 fps). The enemy is PAINT-THROTTLING, not any codec. Chrome correctly stops painting occluded / background / static windows (its battery-saving default), and screencast-style capture is event-driven, so a throttled page silently drops to ~5 fps ("frozen" / sped-up video). Every technique exists to keep frames flowing. (The full cross-surface reference - Hydrogen vs pup, the foreground-rebump trick, page-side animation, the keyframe re-encode - lives in the demo-recording skill's recording-mechanics.md. CDP screencast is NOT garbage - it's pup's browser_record_start engine and does 20-30 fps when the page paints.)

When to use the NATIVE browser (this extension) to record: only when the demo needs the user's real session / cookies / logins (a logged-in tool). If no login is needed, record via Hydrogen or pup (see demo-recording) and don't touch the user's real Chrome.

Native-browser recording = nbrowser_record_start {method:"wgc"|"getdisplaymedia"} (the native path does NOT use CDP screencast - that's pup's). Both are real high-fps engines: wgc (60 fps, Windows, picker-free) and getdisplaymedia (WebRTC/Meet pipeline, 30-60 fps; picker auto-skipped with --auto-select-desktop-capture-source when we launch Chrome).

The key constraint that makes this different from pup: you must NOT foreground-rebump the user's real working browser (pup does that on a throwaway window, but on the user's real Chrome it wrecks their work). Instead, make the real Chrome paint while occluded by launching it with the occlusion/throttle-disable flags --disable-features=CalculateNativeWinOcclusion --disable-backgrounding-occluded-windows --disable-renderer-backgrounding --disable-background-timer-throttling. Default Chrome lacks them (correctly

  • battery). So for a logged-in background recording:
  1. Check whether native Chrome already runs with the flags.
  2. If not → ASK + WARN the user we must fully restart their Chrome, then kill the entire chrome.exe process tree (many helper/background processes - closing the window is NOT enough; the flags don't apply until every chrome.exe is gone), wait for full shutdown, and relaunch with their real profile (logins survive; tabs restore) plus the flags.
  3. Record full-framerate in the background while they keep working.
  4. Later OFFER TO REVERT (battery): "Chrome's in full-background recording mode - back to normal?" → if yes, fully restart Chrome again without the flags. Real session + amazing recordings + normal battery.

Hard requirement under both transitions: a robust full-restart of native Chrome (kill the whole tree, confirm down, relaunch with profile + flags - and again to revert).

⛔ NEVER: foreground the user's real Chrome to fix framerate (use the flags) · leave the flags on globally/permanently (battery drain) · upload a raw recording to the wiki without the -g 30 -keyint_min 30 keyframe re-encode (see recording-mechanics.md).

Captions (two systems)

  • In-page nbrowser_caption {text, style, position} - style:"subtitle" (calm white-on-dark, the HD/web-hydrogen look) or "alert" (red pulsing banner). Tied to a page (cleared on navigation).
  • Fullscreen AD desktop_caption {text, position:"bottom", id} - Win32 always-on-top over every app, truly unmissable; auto-expires. (position:"bottom" - top covers the URL bar + the debugger banner.)

⭐ The killer use case: chip-fetcher through the native browser

EE CAD sources (SnapEDA/SnapMagic, Ultralibrarian, Mouser, DigiKey) gate the actual download behind a login and often a captcha. pup's empty controlled-Chromium hits the wall. The native browser uses the engineer's real logged-in session, so the STEP / KiCad symbol+footprint / Altium / Fusion .lbr bundles just download. Drive the chip-fetcher sourcing ladder (manufacturer-first → SnapMagic → Mouser → DigiKey → … → LCSC last) with nbrowser_* instead of browser_* - same flow, real session, no wall. This is what changes the EE workflow.

Pitfalls + best practices

  • Confirm connected first (nbrowser_status). After dev_reload OR an AD restart, the native host re-launches and re-dials the bridge - expect a reconnect delay (see RECONNECT below).
  • The extension dies if its profile has NO open windows (MV3 SW sleeps) → "not connected". Open a window in that profile; never close your LAST window there.
  • Always clean up windows you open (nbrowser_close_window) - abandoned windows annoy the user, and closing clears that window's taskbar status.
  • Poll nbrowser_events to learn if the USER closed your windows/tabs (window_closed{byUser:true}).
  • nbrowser_eval runs in the PAGE world - chrome.* is NOT available there (only DOM/window).
  • fetch_url is the authed shortcut - real cookies; grabs a file behind a login without clicking.
  • Run nbrowser_help anytime - the bridge self-documents every verb + the flash/taskbar playbook.
  • Driving AD itself: prefer the first-class file verbs (read_file/write_file/list_dir/ delete_file, process_list, net_stats, registry_*) over shell_execute - cleaner, no quoting. (AD 1.8.165 fixed shell_execute's quote-mangling, so quoted Windows paths with spaces now work too - the old ADOMDE~1 8.3-short-name workaround is no longer needed.)

RECONNECT (the one operational gotcha)

Each bridge restart (deploying new bridge code) or AD restart drops the extension; the native host then re-dials and reconnects (currently 30-60s - being fixed AD-side with a stable host port). If nbrowser_status says "not connected" but you didn't change anything, AD probably restarted to deploy a fix - wait, then retry. If it persists, check process_list for stray bridge instances (kill all but one) and that bridge_list runtimePort matches the live bridge's port.

Deploying a code change to the laptop

The extension/ and bridge/ here are source - they run from the laptop, so an edit takes effect only after you push it there and reload:

  • Extension (src/*.js, manifest.json) loads unpacked from C:\Users\john\adom-browser-extension\extension:
    1. write_file each changed file there (real quoted Windows paths work as of AD 1.8.165).
    2. nbrowser_dev_reload - self-completing 2-pass: it does a soft reload, then arms a one-shot flag so the SW does a second hard reload on restart that re-reads disk. A single call now reliably picks up your changes (older builds needed two manual calls because the soft reload served cached files). Throttled 1/90s. Verified: a fresh version loads in ~7s under clean conditions. Gotcha: if you ALSO restart the bridge in the same beat, the SW's reconnect churn can delay the 2nd pass - wait ~10s and don't read nbrowser_status until it settles (the version eventually lands; it's not stuck).
    3. Verify: nbrowser_status → version should reflect your manifest.json bump (bump it so you can tell).
  • Bridge (server.js, bridge.json) runs from …\bridges-cache\native-browser\ - the **single** runtime (bridge_list shows each bridge's entrypoint/source). write_file the changes there, then restart it; if it's a wiki-managed (source:cache) install, keep BRIDGE_VERSION/.bridge-version in sync with the deployed code so a future re-stream stays a no-op.

Managing the extension's lifecycle / driving chrome:// pages → see driving-the-desktop

CDP/nbrowser_* cannot reach chrome:///edge:// pages, native OS dialogs, browser chrome, or an extension that isn't loaded yet - so the extension's own install ("Load unpacked") / uninstall ("Remove") / reload, plus anything on edge://extensions, is driven through AD's desktop_* verbs instead. That whole playbook - the zero-math image-space click path (coordMap + space:"image"), background UI Automation (desktop_find_control → desktop_ui_click/desktop_ui_set, no foreground), full-screen capture that SEES popup bubbles, modifier chords, the ghost-cursor TTL, and the proven uninstall→reinstall recipe (validated on John's Edge 2026-06-23, AD 1.8.175) - lives in the driving-the-desktop sub-skill. Reach for it whenever the user wants to install/uninstall/reload the extension or you need to touch a surface CDP can't.

Install / publish

Extension is loaded unpacked into the user's profile (Chrome 137+ removed --load-extension, so it's a one-time chrome://extensions → Developer mode → Load unpacked; production = Chrome Web Store force-install). Bridge: bridge_install '{"manifestUrl":"…native-browser-bridge-manifest.json"}' → bridges-cache; native-messaging host registers in HKCU (Chrome + Edge); persistent:true keeps one bridge alive on a stable port. See install/, AD-REQUESTS.md, TASKBAR-FEATURE.md.