Adom Browser Extension

// 1) capture →  2) read coordMap.shotId  →  3) open the safe PNG  →  4) click the pixel you see
desktop_click { "space":"image", "shotId":"shot-w656032-7", "x":502, "y":365 }

desktop_navigate {hwnd:<W>, url:"edge://extensions"}   # SetValue omnibox + posted Enter - NO foreground

---
name: driving-the-desktop
description: How to drive the OS desktop with adom-desktop (AD) `desktop_*` verbs when CDP/`nbrowser_*` CAN'T reach the target - the extension's OWN lifecycle (install via "Load unpacked", uninstall via "Remove", reload), `chrome://extensions` / `edge://extensions` and other `chrome://`/`edge://` pages, native file/permission dialogs, the omnibox/toolbar/tabs, and borderless popup bubbles. Match this when the user says "uninstall/reinstall/reload the extension", "install it for me", "the extension isn't loaded", "click the Remove/Load-unpacked button", "drive chrome://…", "click that OS dialog / popup / dropdown", or any browser-chrome UI outside the page. Covers the zero-math image-space click path (coordMap + space:"image"), background UI Automation (find_control → ui_click/ui_set), full-screen capture that SEES popups, modifier chords, and the hard-won traps. Needs AD ≥ 1.8.176 on the laptop (for `desktop_navigate` + `scope:"popups"` - the full background `chrome://` lifecycle) + matching CLI; `--target` the laptop.
---
Parent skill: **driving-the-extension**

When CDP/`nbrowser_*` can't reach something, drive the **OS** instead. This skill is the playbook a
previous AI spent an hour re-discovering - follow it and you won't have to.

> ✅ **The full `chrome://` lifecycle is background as of AD 1.8.176** (verified end-to-end on a real
> Edge uninstall→reinstall, 2026-06-23). `desktop_navigate` opens `chrome://`/`edge://` pages with no
> foreground, and `scope:"popups"` reaches the confirm bubble. **The ONE unavoidable foreground beat** is a
> brief (~1-2s) `desktop_bring_to_front` *while the confirm bubble is up* - the bubble is a Chromium popup
> that dies on focus loss, so the window must hold focus until you click its button. Everything else -
> navigate, find, the Remove/Load-unpacked clicks, even closing the window - is background.

## When CDP fails → switch to desktop control
- **`chrome://` / `edge://` pages** (incl. `…/extensions`, settings) are **blocked to CDP** /
  `nbrowser_navigate`. Only real OS input reaches them.
- **The extension isn't loaded yet** (fresh install / after a Remove) - there's no service worker to talk
  to, so `nbrowser_*` has nothing to drive. Install it via the browser UI.
- **Native OS dialogs** (file pickers, permission prompts) and **browser chrome** (omnibox, toolbar,
  tabs, extension-action **popup bubbles**) live *outside* the page - CDP never sees them.

All verbs below are AD `desktop_*` verbs. **Pass `--target <laptop>`** (e.g. `AdomLapper`); need AD
**≥ 1.8.175** on the box and a matching CLI (`adom-desktop --version`; upgrade with
`ADOMPKG_REGISTRY=https://wiki.adom.inc adompkg install adom/adom-desktop`).

## Capture - and `coordMap` is your map
- **`desktop_screenshot_window {hwnd}`** - PrintWindow of ONE window; **background-safe** (captures
  occluded/background windows, no foreground change). ⚠️ **CANNOT see borderless popups** (extension
  confirm bubbles, native dropdowns) - they're separate windows, absent from the page *and* from
  `desktop_list_windows`.
- **`desktop_screenshot_screen {}`** - FULL virtual-screen grab; **SEES popups/bubbles/dropdowns**. Reach
  for this the moment an action might spawn a confirm bubble.
- Every capture returns, at the **TOP level** (sibling of `output`, not inside it), a **`coordMap`** bound
  to the **delivered safe PNG** (the file at `localSafePath`):
  ```jsonc
  "coordMap": { "shotId":"shot-w<hwnd>-N", "image":{"w":1400,"h":840},
    "screenRect":{"x":-11,"y":-11,"w":2582,"h":1550}, "dpiScale":1.5,
    "imageToScreen":{"sx":1.84,"sy":1.85,"ox":-11,"oy":-11} }   // screenX = imgX*sx+ox
  ```
  Read `shotId`. `screenRect` is in the **same physical-px space** the input verbs + `desktop_list_windows`
  rects use.

## ⭐ Input: click what you SEE - zero math
Pass image-space coords; AD converts server-side. **Never do scale/DPI/rect math again.**
```jsonc
// 1) capture →  2) read coordMap.shotId  →  3) open the safe PNG  →  4) click the pixel you see
desktop_click { "space":"image", "shotId":"shot-w656032-7", "x":502, "y":365 }
```
- Works on `desktop_click` / `double_click` / `right_click` / `hover`. Default `space:"screen"` (physical
  px) is unchanged if you ever pass raw coords.
- `shotId`s: last ~64 retained. `errorCode:"unknown_shot"` → just re-capture for a fresh one.
- **Keyboard:** `desktop_type {hwnd,text}`; `desktop_press_key {keys:[…]}`. **Chords:** `["ctrl","w"]` or
  `["ctrl+w"]` (Ctrl held + W). **Sequence of chords:** `["ctrl+l","enter"]`. Single: `["enter"]`,
  `["f5"]`. The reply echoes `chords` so you can see how AD parsed `keys`.

## ⭐ Background input (no foreground, no focus steal, no cursor move) - PREFER when reachable
Drive controls through the **accessibility tree** - the window is never raised.
- **`desktop_find_control {hwnd|window, name|contains|automationId|role}`** - **READ-ONLY, ungated.**
  Returns `best:{ name, automationId, role, rect:{x,y,w,h,centerX,centerY}, invokable, settable }`
  (rect in **screen px**). **Probe FIRST.**
  - `invokable:true` → **`desktop_ui_click {hwnd, name|contains, role}`** - background Invoke.
  - `settable:true` → **`desktop_ui_set {hwnd, …, text}`** - background SetValue (e.g. the omnibox,
    `name:"Address and search bar"`).
  - both `false` (shadow DOM / not exposed) → SendInput fallback: `desktop_click {x:rect.centerX,
    y:rect.centerY}` (this one foregrounds).
- ⚠️ **Chromium page-content a11y is LAZY.** The **frame** (window buttons, omnibox, tabs) is always in
  the tree, but `chrome://` **page** controls (a card's "Remove") may be **absent on a cold page** -
  Chromium builds the content tree only once it detects an assistive-tech client. **Pattern:**
  `find_control` → if found+invokable, `ui_click`; if `uia_not_found`/only frame controls, **retry once**
  (the query usually wakes the tree); still nothing → SendInput fallback.
- `role ∈ button edit link checkbox menuitem tab listitem combobox radiobutton document group spinner`.
  **Prefer `automationId`** when present - stable + locale-independent.

## Ghost cursor
`cursor:true` + `intent:"…"` on a click verb glides the teal Adom cursor (a trust tell). As of AD 1.8.175
it **auto-hides after 60s** (refreshed on each show/move). Extend with `ttlMs`, or `persist:true` then
hide yourself. **No need to hide "to be safe"** - only hide early to clear it sooner.

## Recipe - open a `chrome://`/`edge://` page (CDP can't) - BACKGROUND (AD 1.8.176)
Open a background window first (`nbrowser_open_window {sessionId:"drive", thread:"<you>", purpose:"<why>", profile, url:"https://example.com"}`,
then find its `hwnd`), then:
```
desktop_navigate {hwnd:<W>, url:"edge://extensions"}   # SetValue omnibox + posted Enter - NO foreground
```
⚠️ **Scheme must match the browser:** `chrome://…` for Chrome, `edge://…` for Edge. Give Chrome an
`edge://` URL (or vice-versa) and the omnibox **searches** the text instead of navigating. `https://` works
on either. **Always verify** with `desktop_screenshot_window` that the PAGE actually loaded (the target's
tab title changes, the page renders) - NOT just that the omnibox shows the URL; the posted Enter sometimes
does not commit and you must re-issue `desktop_navigate`. Fallback if a target ignores it:
`desktop_bring_to_front` + `desktop_press_key ["enter"]` (brief foreground).
(`desktop_ui_set` also takes `submit:true` to set+commit an arbitrary field.)
⚠️ **`desktop_navigate` is NOT always foreground-free.** Despite the "NO foreground" note above, its
omnibox commit can **surface/activate the window** (observed: a `chrome://extensions` navigate left the
window at `z:0` with no `bring_to_front` and `focused:false`). Foreground is about the OUTCOME, not the
verb: after a navigate, check `desktop_list_windows` (`z:0` = topmost = you surfaced it), and avoid
repeated navigations while the user is mid-task.

## Recipe - UNINSTALL the extension - BACKGROUND except one ~1s focus beat (verified AD 1.8.176)
1. `desktop_navigate {hwnd:<W>, url:"edge://extensions"}` (background). `find_control {hwnd:<W>,
   contains:"<ext name>"}` to confirm the card is present.
2. The confirm bubble is a Chromium **owned popup** that dies on focus loss, so it must hold focus while you
   click it. The working atomic sequence - **nothing between the trigger and the confirm**:
   ```
   desktop_bring_to_front {hwnd:<W>}                                  # the ONE ~1s foreground beat
   desktop_ui_click {hwnd:<W>, role:"button", contains:"Remove"}      # trigger → opens the bubble
   # wait ~1s
   desktop_ui_click {hwnd:<W>, scope:"popups", role:"button", name:"Remove"}   # confirm (background Invoke)
   ```
   `scope:"popups"` enumerates the window's **owned, borderless** popups (absent from `desktop_list_windows`)
   and UIA-searches them; the bubble's window title is `Remove "<ext>" from <browser>?`. **Don't** insert a
   `screenshot`/extra `find_control` between trigger and confirm - the bubble (~1-2s lifetime) closes and
   you'll get `scope:popups found no owned/popup windows`; just re-trigger and confirm immediately.
3. **Verify by state:** poll `nbrowser_profiles` - the `edge:`/`chrome:` key **drops** when the SW dies.

## Recipe - REINSTALL (Load unpacked) - BACKGROUND
1. `desktop_ui_click {hwnd:<W>, contains:"Load unpacked"}` → opens a Win32 **folder picker** (`#32770`;
   enumerates in `desktop_list_windows`, has an a11y tree).
2. `desktop_ui_click {hwnd:<picker>, role:"button", contains:"Select"}` to confirm the folder.
   ⚠️ `desktop_ui_set` on the picker's edit field can return *"operation was canceled by user"* - but the
   picker **remembers the last-used dir**, so if it's already inside the extension folder, Select Folder
   grabs the right one. If it might open elsewhere, navigate the picker / set the path some other way first
   (open AD ask: reliable picker path-fill).
3. Load-unpacked reads the **on-disk** build (version may jump). Verify the `edge:`/`chrome:` key returns to
   `nbrowser_profiles` (reconnects in ~1-2s).

## ⛔⛔ NEVER take over a PWA / installed-app window (first-class guard, ext 0.0.14+)
Installed PWAs / "Add to taskbar" apps - **Google Chat, Google Messages, Gmail-as-app, etc.** - run in
their OWN window, which Chrome reports as **`type:'app'`** (vs `'normal'` for ordinary browsing). If you
let a drive/navigate verb fall back to the user's **currently-focused** window and that window is a PWA,
you **hijack their standalone app** (real regression 2026-06-24: a `nbrowser_navigate` with an unmapped
session fell back to `lastFocusedWindow` - the **Google Chat PWA** - and navigated it to Gusto).
- **The extension now refuses this at the choke point** (`resolveTabId` → `assertDriveableWindow`): any
  verb that resolves to a non-`'normal'` window throws *"refusing to drive a PWA / installed-app window…"*
  instead of acting. So **always target a window you opened** with `nbrowser_open_window` (those are
  `'normal'`), never assume the focused window is yours.
- **Override (only when the user explicitly asks to drive that exact app window):** pass **`allowPwa:true`**.
- Note: **READING** a PWA via UIA (e.g. `reading-google-messages`) is fine - that doesn't navigate it. The
  ban is on **driving/navigating** an app window.

## ⛔ You CANNOT decrypt modern Chrome/Edge saved passwords (v20 app-bound encryption)
Don't build a "decrypt the saved password + inject it" autologin - it's a dead end for modern browsers.
Chrome/Edge passwords saved since ~Chrome 127 (2024) use the **`v20`** prefix = **app-bound encryption**,
an anti-infostealer feature: the key is bound to **chrome.exe's own identity** (unwrapped only via Chrome's
elevation COM service, validated against its code signature). **No separate process - bridge, native host,
or otherwise - can decrypt them**, even as the same user in the same session (verified 2026-06-24 on Gusto;
the host decrypts the legacy `os_crypt` key fine, `keylen=32`, but that key can't touch `v20` blobs).
- Legacy `v10`/`v11` blobs ARE decryptable (DPAPI os_crypt key + AES-GCM), but almost everything is `v20` now.
- **The only way to USE a saved password is to let the browser fill it** - drive Chrome's native autofill
  dropdown (Chrome does the `v20` decryption itself). That dropdown is an OS popup that Chrome shows ONLY
  for the **foreground** window on a real gesture, so a ~1s foreground to summon it is unavoidable; then
  drive it via the owned-popup screenshot (`desktop_screenshot_window` → `screenshots[]`, AD 1.8.177+) +
  `desktop_click {space:"image",shotId,...}` or `desktop_ui_click {scope:"popups"}`. A CDP click does NOT
  summon it (Chrome requires a real OS gesture + foreground window). **Fully-background password fill is
  thus impossible for app-bound passwords - a Chrome boundary, not an Adom one.**

### ✅ Recipe - log in with a saved password (autofill via owned-popup, PROVEN on Gusto 2026-06-24)
The working flow. Mostly background; the ONLY foreground is the ~1s summon.
1. **Background:** `nbrowser_*` navigate to the login page; on a split/Keycloak form (e.g. Gusto), `nbrowser_type`
   the email + `nbrowser_click` "Continue" (email isn't secret) → land on the password step. Confirm with
   `nbrowser_eval "!!document.querySelector('input[type=password]')"` (background - NOT a fullscreen grab).
2. **Find the field:** `desktop_find_control {hwnd, automationId:"password"}` → its screen `rect.centerX/Y`.
   (⚠️ `role:"edit"` alone matches the OMNIBOX first - target by `automationId:"password"`.)
3. **Summon (the ~1s foreground):** `desktop_bring_to_front {hwnd}` → `desktop_click {hwnd, x:centerX, y:centerY}`
   (a REAL OS click - Chrome only shows the autofill popup for the foreground window on a real gesture).
4. **See it (background-safe):** `desktop_screenshot_window {hwnd}` → `ownedPopupCount` should be `1` and
   `screenshots[0]` is the autofill dropdown (its own PNG + its own `shotId`). Read its `localSafePath`; the
   saved login shows as "<email>" + a row of dots.
5. **Pick it:** `desktop_click {space:"image", shotId:<that popup's shotId>, x, y}` on the saved-login row
   (x,y = pixels in the POPUP image; the row is in the upper third). Chrome fills the password - verify
   `nbrowser_eval` on `input[type=password]`.value.length > 0. (`desktop_press_key` arrows did NOT drive the
   popup - clicking the row by its own shotId is the reliable mechanism, and you don't have to count downs.)
6. **Background:** `nbrowser_click` "Continue"/"Sign in" → verify via `location.href` (left the login host) +
   no `input[type=password]` + no 2FA text. Handle a 2FA gate by surfacing it (it needs a human code).

## ⚠️ THE CDP DEBUGGER BANNER SHIFTS PAGE COORDS (sabotages desktop clicks)
When the extension attaches `chrome.debugger` (on the FIRST `nbrowser_eval`/`screenshot`/`click` of a
tab), Chrome slides in the **"Adom for Chrome & Edge started debugging this browser"** banner. It pushes
the **page content down ~84px (screen) / ~58 CSS px** - but NOT the toolbar/omnibox. Measured live
2026-06-23: page document top = 150px (banner off) → 234px (banner on).

**The self-sabotage pattern** (cost hours on the Gusto autologin): `find_control` measures a page control's
screen-y with the banner OFF (debugger idle-detached after 8s) → your NEXT step is a CDP call → the banner
slides in → the page drops 84px → your `desktop_click` at the old y lands 84px HIGH and misses.

**Rules:**
- **Don't mix a CDP call between measuring screen coords and clicking.** If you must, re-measure AFTER.
- **Settle the banner first:** do a throwaway CDP op (`nbrowser_eval {expr:"1"}`) so the banner is UP, wait
  ~1s, THEN `find_control` + `desktop_click` (banner stable through both). The debugger idle-detaches after
  8s, so the banner state flips under you if you pause.
- **Best: avoid screen coords for browser PAGE content entirely.** Page elements are driven banner-immune by
  CDP (`nbrowser_click`/`type` use page coords, unaffected by the banner). Reserve `desktop_click` for
  browser CHROME (toolbar, OS popups) which the banner doesn't move.
- On Adom-managed machines `--silent-debugger-extension-api` removes the banner - then the shift is gone.

## Why these traps exist (so you don't re-discover them)
- **"My click did nothing"** was an illusion - the click worked; the confirm bubble was just *invisible to
  window-capture* (it's an owned popup). Reach it with `find_control`/`ui_click` `scope:"popups"`; to merely
  see it, `screenshot_screen`.
- **The confirm bubble lives ~1-2s and dies on focus loss.** Hold focus with `desktop_bring_to_front`,
  then trigger + `scope:"popups"` confirm with NOTHING in between (no screenshot/extra find_control).
- **SendInput foregrounds** (lands only on the foreground window) → interrupts the user. Prefer UIA
  (`ui_click`/`ui_set`/`desktop_navigate`) which is background; reserve SendInput for controls UIA can't
  reach.
- **Don't pixel-hunt.** Image-space click (`space:"image"`) or `find_control` rect. Don't fetch the window
  rect to compute scale - `coordMap` already has it.

Canonical `desktop_*` verb reference: https://wiki.adom.inc/adom/adom-desktop