← Commit history
skills/driving-the-desktop/SKILL.md+48−48@@ -1,37 +1,37 @@ --- name: driving-the-desktop-description: How to drive the OS desktop with adom-desktop (AD) `desktop_*` verbs when CDP/`nbrowser_*` CAN'T reach the target — the extension's OWN lifecycle (install via "Load unpacked", uninstall via "Remove", reload), `chrome://extensions` / `edge://extensions` and other `chrome://`/`edge://` pages, native file/permission dialogs, the omnibox/toolbar/tabs, and borderless popup bubbles. Match this when the user says "uninstall/reinstall/reload the extension", "install it for me", "the extension isn't loaded", "click the Remove/Load-unpacked button", "drive chrome://…", "click that OS dialog / popup / dropdown", or any browser-chrome UI outside the page. Covers the zero-math image-space click path (coordMap + space:"image"), background UI Automation (find_control → ui_click/ui_set), full-screen capture that SEES popups, modifier chords, and the hard-won traps. Needs AD ≥ 1.8.176 on the laptop (for `desktop_navigate` + `scope:"popups"` — the full background `chrome://` lifecycle) + matching CLI; `--target` the laptop.+description: How to drive the OS desktop with adom-desktop (AD) `desktop_*` verbs when CDP/`nbrowser_*` CAN'T reach the target - the extension's OWN lifecycle (install via "Load unpacked", uninstall via "Remove", reload), `chrome://extensions` / `edge://extensions` and other `chrome://`/`edge://` pages, native file/permission dialogs, the omnibox/toolbar/tabs, and borderless popup bubbles. Match this when the user says "uninstall/reinstall/reload the extension", "install it for me", "the extension isn't loaded", "click the Remove/Load-unpacked button", "drive chrome://…", "click that OS dialog / popup / dropdown", or any browser-chrome UI outside the page. Covers the zero-math image-space click path (coordMap + space:"image"), background UI Automation (find_control → ui_click/ui_set), full-screen capture that SEES popups, modifier chords, and the hard-won traps. Needs AD ≥ 1.8.176 on the laptop (for `desktop_navigate` + `scope:"popups"` - the full background `chrome://` lifecycle) + matching CLI; `--target` the laptop. --- Parent skill: **driving-the-extension** When CDP/`nbrowser_*` can't reach something, drive the **OS** instead. This skill is the playbook a-previous AI spent an hour re-discovering — follow it and you won't have to.+previous AI spent an hour re-discovering - follow it and you won't have to. > ✅ **The full `chrome://` lifecycle is background as of AD 1.8.176** (verified end-to-end on a real > Edge uninstall→reinstall, 2026-06-23). `desktop_navigate` opens `chrome://`/`edge://` pages with no > foreground, and `scope:"popups"` reaches the confirm bubble. **The ONE unavoidable foreground beat** is a-> brief (~1–2s) `desktop_bring_to_front` *while the confirm bubble is up* — the bubble is a Chromium popup-> that dies on focus loss, so the window must hold focus until you click its button. Everything else —-> navigate, find, the Remove/Load-unpacked clicks, even closing the window — is background.+> brief (~1-2s) `desktop_bring_to_front` *while the confirm bubble is up* - the bubble is a Chromium popup+> that dies on focus loss, so the window must hold focus until you click its button. Everything else -+> navigate, find, the Remove/Load-unpacked clicks, even closing the window - is background. ## When CDP fails → switch to desktop control - **`chrome://` / `edge://` pages** (incl. `…/extensions`, settings) are **blocked to CDP** / `nbrowser_navigate`. Only real OS input reaches them.-- **The extension isn't loaded yet** (fresh install / after a Remove) — there's no service worker to talk+- **The extension isn't loaded yet** (fresh install / after a Remove) - there's no service worker to talk to, so `nbrowser_*` has nothing to drive. Install it via the browser UI. - **Native OS dialogs** (file pickers, permission prompts) and **browser chrome** (omnibox, toolbar,- tabs, extension-action **popup bubbles**) live *outside* the page — CDP never sees them.+ tabs, extension-action **popup bubbles**) live *outside* the page - CDP never sees them. All verbs below are AD `desktop_*` verbs. **Pass `--target <laptop>`** (e.g. `AdomLapper`); need AD **≥ 1.8.175** on the box and a matching CLI (`adom-desktop --version`; upgrade with `ADOMPKG_REGISTRY=https://wiki.adom.inc adompkg install adom/adom-desktop`). -## Capture — and `coordMap` is your map-- **`desktop_screenshot_window {hwnd}`** — PrintWindow of ONE window; **background-safe** (captures+## Capture - and `coordMap` is your map+- **`desktop_screenshot_window {hwnd}`** - PrintWindow of ONE window; **background-safe** (captures occluded/background windows, no foreground change). ⚠️ **CANNOT see borderless popups** (extension- confirm bubbles, native dropdowns) — they're separate windows, absent from the page *and* from+ confirm bubbles, native dropdowns) - they're separate windows, absent from the page *and* from `desktop_list_windows`.-- **`desktop_screenshot_screen {}`** — FULL virtual-screen grab; **SEES popups/bubbles/dropdowns**. Reach+- **`desktop_screenshot_screen {}`** - FULL virtual-screen grab; **SEES popups/bubbles/dropdowns**. Reach for this the moment an action might spawn a confirm bubble. - Every capture returns, at the **TOP level** (sibling of `output`, not inside it), a **`coordMap`** bound to the **delivered safe PNG** (the file at `localSafePath`):@@ -43,7 +43,7 @@ All verbs below are AD `desktop_*` verbs. **Pass `--target <laptop>`** (e.g. `Ad Read `shotId`. `screenRect` is in the **same physical-px space** the input verbs + `desktop_list_windows` rects use. -## ⭐ Input: click what you SEE — zero math+## ⭐ Input: click what you SEE - zero math Pass image-space coords; AD converts server-side. **Never do scale/DPI/rect math again.** ```jsonc // 1) capture → 2) read coordMap.shotId → 3) open the safe PNG → 4) click the pixel you see@@ -56,34 +56,34 @@ desktop_click { "space":"image", "shotId":"shot-w656032-7", "x":502, "y":365 } `["ctrl+w"]` (Ctrl held + W). **Sequence of chords:** `["ctrl+l","enter"]`. Single: `["enter"]`, `["f5"]`. The reply echoes `chords` so you can see how AD parsed `keys`. -## ⭐ Background input (no foreground, no focus steal, no cursor move) — PREFER when reachable-Drive controls through the **accessibility tree** — the window is never raised.-- **`desktop_find_control {hwnd|window, name|contains|automationId|role}`** — **READ-ONLY, ungated.**+## ⭐ Background input (no foreground, no focus steal, no cursor move) - PREFER when reachable+Drive controls through the **accessibility tree** - the window is never raised.+- **`desktop_find_control {hwnd|window, name|contains|automationId|role}`** - **READ-ONLY, ungated.** Returns `best:{ name, automationId, role, rect:{x,y,w,h,centerX,centerY}, invokable, settable }` (rect in **screen px**). **Probe FIRST.**- - `invokable:true` → **`desktop_ui_click {hwnd, name|contains, role}`** — background Invoke.- - `settable:true` → **`desktop_ui_set {hwnd, …, text}`** — background SetValue (e.g. the omnibox,+ - `invokable:true` → **`desktop_ui_click {hwnd, name|contains, role}`** - background Invoke.+ - `settable:true` → **`desktop_ui_set {hwnd, …, text}`** - background SetValue (e.g. the omnibox, `name:"Address and search bar"`). - both `false` (shadow DOM / not exposed) → SendInput fallback: `desktop_click {x:rect.centerX, y:rect.centerY}` (this one foregrounds). - ⚠️ **Chromium page-content a11y is LAZY.** The **frame** (window buttons, omnibox, tabs) is always in- the tree, but `chrome://` **page** controls (a card's "Remove") may be **absent on a cold page** —+ the tree, but `chrome://` **page** controls (a card's "Remove") may be **absent on a cold page** - Chromium builds the content tree only once it detects an assistive-tech client. **Pattern:** `find_control` → if found+invokable, `ui_click`; if `uia_not_found`/only frame controls, **retry once** (the query usually wakes the tree); still nothing → SendInput fallback. - `role ∈ button edit link checkbox menuitem tab listitem combobox radiobutton document group spinner`.- **Prefer `automationId`** when present — stable + locale-independent.+ **Prefer `automationId`** when present - stable + locale-independent. ## Ghost cursor `cursor:true` + `intent:"…"` on a click verb glides the teal Adom cursor (a trust tell). As of AD 1.8.175 it **auto-hides after 60s** (refreshed on each show/move). Extend with `ttlMs`, or `persist:true` then-hide yourself. **No need to hide "to be safe"** — only hide early to clear it sooner.+hide yourself. **No need to hide "to be safe"** - only hide early to clear it sooner. -## Recipe — open a `chrome://`/`edge://` page (CDP can't) — BACKGROUND (AD 1.8.176)+## Recipe - open a `chrome://`/`edge://` page (CDP can't) - BACKGROUND (AD 1.8.176) Open a background window first (`nbrowser_open_window {profile, url:"https://example.com", focused:false}`, then find its `hwnd`), then: ```-desktop_navigate {hwnd:<W>, url:"edge://extensions"} # SetValue omnibox + posted Enter — NO foreground+desktop_navigate {hwnd:<W>, url:"edge://extensions"} # SetValue omnibox + posted Enter - NO foreground ``` ⚠️ **Scheme must match the browser:** `chrome://…` for Chrome, `edge://…` for Edge. Give Chrome an `edge://` URL (or vice-versa) and the omnibox **searches** the text instead of navigating. `https://` works@@ -98,11 +98,11 @@ window at `z:0` with no `bring_to_front` and `focused:false`). Foreground is abo verb: after a navigate, check `desktop_list_windows` (`z:0` = topmost = you surfaced it), and avoid repeated navigations while the user is mid-task. -## Recipe — UNINSTALL the extension — BACKGROUND except one ~1s focus beat (verified AD 1.8.176)+## Recipe - UNINSTALL the extension - BACKGROUND except one ~1s focus beat (verified AD 1.8.176) 1. `desktop_navigate {hwnd:<W>, url:"edge://extensions"}` (background). `find_control {hwnd:<W>, contains:"<ext name>"}` to confirm the card is present. 2. The confirm bubble is a Chromium **owned popup** that dies on focus loss, so it must hold focus while you- click it. The working atomic sequence — **nothing between the trigger and the confirm**:+ click it. The working atomic sequence - **nothing between the trigger and the confirm**: ``` desktop_bring_to_front {hwnd:<W>} # the ONE ~1s foreground beat desktop_ui_click {hwnd:<W>, role:"button", contains:"Remove"} # trigger → opens the bubble@@ -111,74 +111,74 @@ repeated navigations while the user is mid-task. ``` `scope:"popups"` enumerates the window's **owned, borderless** popups (absent from `desktop_list_windows`) and UIA-searches them; the bubble's window title is `Remove "<ext>" from <browser>?`. **Don't** insert a- `screenshot`/extra `find_control` between trigger and confirm — the bubble (~1–2s lifetime) closes and+ `screenshot`/extra `find_control` between trigger and confirm - the bubble (~1-2s lifetime) closes and you'll get `scope:popups found no owned/popup windows`; just re-trigger and confirm immediately.-3. **Verify by state:** poll `nbrowser_profiles` — the `edge:`/`chrome:` key **drops** when the SW dies.+3. **Verify by state:** poll `nbrowser_profiles` - the `edge:`/`chrome:` key **drops** when the SW dies. -## Recipe — REINSTALL (Load unpacked) — BACKGROUND+## Recipe - REINSTALL (Load unpacked) - BACKGROUND 1. `desktop_ui_click {hwnd:<W>, contains:"Load unpacked"}` → opens a Win32 **folder picker** (`#32770`; enumerates in `desktop_list_windows`, has an a11y tree). 2. `desktop_ui_click {hwnd:<picker>, role:"button", contains:"Select"}` to confirm the folder.- ⚠️ `desktop_ui_set` on the picker's edit field can return *"operation was canceled by user"* — but the+ ⚠️ `desktop_ui_set` on the picker's edit field can return *"operation was canceled by user"* - but the picker **remembers the last-used dir**, so if it's already inside the extension folder, Select Folder grabs the right one. If it might open elsewhere, navigate the picker / set the path some other way first (open AD ask: reliable picker path-fill). 3. Load-unpacked reads the **on-disk** build (version may jump). Verify the `edge:`/`chrome:` key returns to- `nbrowser_profiles` (reconnects in ~1–2s).+ `nbrowser_profiles` (reconnects in ~1-2s). ## ⛔⛔ NEVER take over a PWA / installed-app window (first-class guard, ext 0.0.14+)-Installed PWAs / "Add to taskbar" apps — **Google Chat, Google Messages, Gmail-as-app, etc.** — run in+Installed PWAs / "Add to taskbar" apps - **Google Chat, Google Messages, Gmail-as-app, etc.** - run in their OWN window, which Chrome reports as **`type:'app'`** (vs `'normal'` for ordinary browsing). If you let a drive/navigate verb fall back to the user's **currently-focused** window and that window is a PWA, you **hijack their standalone app** (real regression 2026-06-24: a `nbrowser_navigate` with an unmapped-session fell back to `lastFocusedWindow` — the **Google Chat PWA** — and navigated it to Gusto).+session fell back to `lastFocusedWindow` - the **Google Chat PWA** - and navigated it to Gusto). - **The extension now refuses this at the choke point** (`resolveTabId` → `assertDriveableWindow`): any verb that resolves to a non-`'normal'` window throws *"refusing to drive a PWA / installed-app window…"* instead of acting. So **always target a window you opened** with `nbrowser_open_window` (those are `'normal'`), never assume the focused window is yours. - **Override (only when the user explicitly asks to drive that exact app window):** pass **`allowPwa:true`**.-- Note: **READING** a PWA via UIA (e.g. `reading-google-messages`) is fine — that doesn't navigate it. The+- Note: **READING** a PWA via UIA (e.g. `reading-google-messages`) is fine - that doesn't navigate it. The ban is on **driving/navigating** an app window. ## ⛔ You CANNOT decrypt modern Chrome/Edge saved passwords (v20 app-bound encryption)-Don't build a "decrypt the saved password + inject it" autologin — it's a dead end for modern browsers.+Don't build a "decrypt the saved password + inject it" autologin - it's a dead end for modern browsers. Chrome/Edge passwords saved since ~Chrome 127 (2024) use the **`v20`** prefix = **app-bound encryption**, an anti-infostealer feature: the key is bound to **chrome.exe's own identity** (unwrapped only via Chrome's-elevation COM service, validated against its code signature). **No separate process — bridge, native host,-or otherwise — can decrypt them**, even as the same user in the same session (verified 2026-06-24 on Gusto;+elevation COM service, validated against its code signature). **No separate process - bridge, native host,+or otherwise - can decrypt them**, even as the same user in the same session (verified 2026-06-24 on Gusto; the host decrypts the legacy `os_crypt` key fine, `keylen=32`, but that key can't touch `v20` blobs). - Legacy `v10`/`v11` blobs ARE decryptable (DPAPI os_crypt key + AES-GCM), but almost everything is `v20` now.-- **The only way to USE a saved password is to let the browser fill it** — drive Chrome's native autofill+- **The only way to USE a saved password is to let the browser fill it** - drive Chrome's native autofill dropdown (Chrome does the `v20` decryption itself). That dropdown is an OS popup that Chrome shows ONLY for the **foreground** window on a real gesture, so a ~1s foreground to summon it is unavoidable; then drive it via the owned-popup screenshot (`desktop_screenshot_window` → `screenshots[]`, AD 1.8.177+) + `desktop_click {space:"image",shotId,...}` or `desktop_ui_click {scope:"popups"}`. A CDP click does NOT summon it (Chrome requires a real OS gesture + foreground window). **Fully-background password fill is- thus impossible for app-bound passwords — a Chrome boundary, not an Adom one.**+ thus impossible for app-bound passwords - a Chrome boundary, not an Adom one.** -### ✅ Recipe — log in with a saved password (autofill via owned-popup, PROVEN on Gusto 2026-06-24)+### ✅ Recipe - log in with a saved password (autofill via owned-popup, PROVEN on Gusto 2026-06-24) The working flow. Mostly background; the ONLY foreground is the ~1s summon. 1. **Background:** `nbrowser_*` navigate to the login page; on a split/Keycloak form (e.g. Gusto), `nbrowser_type` the email + `nbrowser_click` "Continue" (email isn't secret) → land on the password step. Confirm with- `nbrowser_eval "!!document.querySelector('input[type=password]')"` (background — NOT a fullscreen grab).+ `nbrowser_eval "!!document.querySelector('input[type=password]')"` (background - NOT a fullscreen grab). 2. **Find the field:** `desktop_find_control {hwnd, automationId:"password"}` → its screen `rect.centerX/Y`.- (⚠️ `role:"edit"` alone matches the OMNIBOX first — target by `automationId:"password"`.)+ (⚠️ `role:"edit"` alone matches the OMNIBOX first - target by `automationId:"password"`.) 3. **Summon (the ~1s foreground):** `desktop_bring_to_front {hwnd}` → `desktop_click {hwnd, x:centerX, y:centerY}`- (a REAL OS click — Chrome only shows the autofill popup for the foreground window on a real gesture).+ (a REAL OS click - Chrome only shows the autofill popup for the foreground window on a real gesture). 4. **See it (background-safe):** `desktop_screenshot_window {hwnd}` → `ownedPopupCount` should be `1` and `screenshots[0]` is the autofill dropdown (its own PNG + its own `shotId`). Read its `localSafePath`; the saved login shows as "<email>" + a row of dots. 5. **Pick it:** `desktop_click {space:"image", shotId:<that popup's shotId>, x, y}` on the saved-login row- (x,y = pixels in the POPUP image; the row is in the upper third). Chrome fills the password — verify+ (x,y = pixels in the POPUP image; the row is in the upper third). Chrome fills the password - verify `nbrowser_eval` on `input[type=password]`.value.length > 0. (`desktop_press_key` arrows did NOT drive the- popup — clicking the row by its own shotId is the reliable mechanism, and you don't have to count downs.)+ popup - clicking the row by its own shotId is the reliable mechanism, and you don't have to count downs.) 6. **Background:** `nbrowser_click` "Continue"/"Sign in" → verify via `location.href` (left the login host) + no `input[type=password]` + no 2FA text. Handle a 2FA gate by surfacing it (it needs a human code). ## ⚠️ THE CDP DEBUGGER BANNER SHIFTS PAGE COORDS (sabotages desktop clicks) When the extension attaches `chrome.debugger` (on the FIRST `nbrowser_eval`/`screenshot`/`click` of a tab), Chrome slides in the **"Adom for Chrome & Edge started debugging this browser"** banner. It pushes-the **page content down ~84px (screen) / ~58 CSS px** — but NOT the toolbar/omnibox. Measured live+the **page content down ~84px (screen) / ~58 CSS px** - but NOT the toolbar/omnibox. Measured live 2026-06-23: page document top = 150px (banner off) → 234px (banner on). **The self-sabotage pattern** (cost hours on the Gusto autologin): `find_control` measures a page control's@@ -193,18 +193,18 @@ slides in → the page drops 84px → your `desktop_click` at the old y lands 84 - **Best: avoid screen coords for browser PAGE content entirely.** Page elements are driven banner-immune by CDP (`nbrowser_click`/`type` use page coords, unaffected by the banner). Reserve `desktop_click` for browser CHROME (toolbar, OS popups) which the banner doesn't move.-- On Adom-managed machines `--silent-debugger-extension-api` removes the banner — then the shift is gone.+- On Adom-managed machines `--silent-debugger-extension-api` removes the banner - then the shift is gone. ## Why these traps exist (so you don't re-discover them)-- **"My click did nothing"** was an illusion — the click worked; the confirm bubble was just *invisible to+- **"My click did nothing"** was an illusion - the click worked; the confirm bubble was just *invisible to window-capture* (it's an owned popup). Reach it with `find_control`/`ui_click` `scope:"popups"`; to merely see it, `screenshot_screen`.-- **The confirm bubble lives ~1–2s and dies on focus loss.** Hold focus with `desktop_bring_to_front`,+- **The confirm bubble lives ~1-2s and dies on focus loss.** Hold focus with `desktop_bring_to_front`, then trigger + `scope:"popups"` confirm with NOTHING in between (no screenshot/extra find_control). - **SendInput foregrounds** (lands only on the foreground window) → interrupts the user. Prefer UIA (`ui_click`/`ui_set`/`desktop_navigate`) which is background; reserve SendInput for controls UIA can't reach. - **Don't pixel-hunt.** Image-space click (`space:"image"`) or `find_control` rect. Don't fetch the window- rect to compute scale — `coordMap` already has it.+ rect to compute scale - `coordMap` already has it. Canonical `desktop_*` verb reference: https://wiki.adom.inc/adom/adom-desktop