NB - Native Browser Bridge

@@ -0,0 +1,116 @@+---+name: driving-the-desktop+description: How to drive the OS desktop with adom-desktop (AD) `desktop_*` verbs when CDP/`nbrowser_*` CAN'T reach the target — the extension's OWN lifecycle (install via "Load unpacked", uninstall via "Remove", reload), `chrome://extensions` / `edge://extensions` and other `chrome://`/`edge://` pages, native file/permission dialogs, the omnibox/toolbar/tabs, and borderless popup bubbles. Match this when the user says "uninstall/reinstall/reload the extension", "install it for me", "the extension isn't loaded", "click the Remove/Load-unpacked button", "drive chrome://…", "click that OS dialog / popup / dropdown", or any browser-chrome UI outside the page. Covers the zero-math image-space click path (coordMap + space:"image"), background UI Automation (find_control → ui_click/ui_set), full-screen capture that SEES popups, modifier chords, and the hard-won traps. Needs AD ≥ 1.8.175 on the laptop + matching CLI; `--target` the laptop.+---+Parent skill: **driving-the-extension**++When CDP/`nbrowser_*` can't reach something, drive the **OS** instead. This skill is the playbook a+previous AI spent an hour re-discovering — follow it and you won't have to.++## When CDP fails → switch to desktop control+- **`chrome://` / `edge://` pages** (incl. `…/extensions`, settings) are **blocked to CDP** /+  `nbrowser_navigate`. Only real OS input reaches them.+- **The extension isn't loaded yet** (fresh install / after a Remove) — there's no service worker to talk+  to, so `nbrowser_*` has nothing to drive. Install it via the browser UI.+- **Native OS dialogs** (file pickers, permission prompts) and **browser chrome** (omnibox, toolbar,+  tabs, extension-action **popup bubbles**) live *outside* the page — CDP never sees them.++All verbs below are AD `desktop_*` verbs. **Pass `--target <laptop>`** (e.g. `AdomLapper`); need AD+**≥ 1.8.175** on the box and a matching CLI (`adom-desktop --version`; upgrade with+`ADOMPKG_REGISTRY=https://wiki.adom.inc adompkg install adom/adom-desktop`).++## Capture — and `coordMap` is your map+- **`desktop_screenshot_window {hwnd}`** — PrintWindow of ONE window; **background-safe** (captures+  occluded/background windows, no foreground change). ⚠️ **CANNOT see borderless popups** (extension+  confirm bubbles, native dropdowns) — they're separate windows, absent from the page *and* from+  `desktop_list_windows`.+- **`desktop_screenshot_screen {}`** — FULL virtual-screen grab; **SEES popups/bubbles/dropdowns**. Reach+  for this the moment an action might spawn a confirm bubble.+- Every capture returns, at the **TOP level** (sibling of `output`, not inside it), a **`coordMap`** bound+  to the **delivered safe PNG** (the file at `localSafePath`):+  ```jsonc+  "coordMap": { "shotId":"shot-w<hwnd>-N", "image":{"w":1400,"h":840},+    "screenRect":{"x":-11,"y":-11,"w":2582,"h":1550}, "dpiScale":1.5,+    "imageToScreen":{"sx":1.84,"sy":1.85,"ox":-11,"oy":-11} }   // screenX = imgX*sx+ox+  ```+  Read `shotId`. `screenRect` is in the **same physical-px space** the input verbs + `desktop_list_windows`+  rects use.++## ⭐ Input: click what you SEE — zero math+Pass image-space coords; AD converts server-side. **Never do scale/DPI/rect math again.**+```jsonc+// 1) capture →  2) read coordMap.shotId  →  3) open the safe PNG  →  4) click the pixel you see+desktop_click { "space":"image", "shotId":"shot-w656032-7", "x":502, "y":365 }+```+- Works on `desktop_click` / `double_click` / `right_click` / `hover`. Default `space:"screen"` (physical+  px) is unchanged if you ever pass raw coords.+- `shotId`s: last ~64 retained. `errorCode:"unknown_shot"` → just re-capture for a fresh one.+- **Keyboard:** `desktop_type {hwnd,text}`; `desktop_press_key {keys:[…]}`. **Chords:** `["ctrl","w"]` or+  `["ctrl+w"]` (Ctrl held + W). **Sequence of chords:** `["ctrl+l","enter"]`. Single: `["enter"]`,+  `["f5"]`. The reply echoes `chords` so you can see how AD parsed `keys`.++## ⭐ Background input (no foreground, no focus steal, no cursor move) — PREFER when reachable+Drive controls through the **accessibility tree** — the window is never raised.+- **`desktop_find_control {hwnd|window, name|contains|automationId|role}`** — **READ-ONLY, ungated.**+  Returns `best:{ name, automationId, role, rect:{x,y,w,h,centerX,centerY}, invokable, settable }`+  (rect in **screen px**). **Probe FIRST.**+  - `invokable:true` → **`desktop_ui_click {hwnd, name|contains, role}`** — background Invoke.+  - `settable:true` → **`desktop_ui_set {hwnd, …, text}`** — background SetValue (e.g. the omnibox,+    `name:"Address and search bar"`).+  - both `false` (shadow DOM / not exposed) → SendInput fallback: `desktop_click {x:rect.centerX,+    y:rect.centerY}` (this one foregrounds).+- ⚠️ **Chromium page-content a11y is LAZY.** The **frame** (window buttons, omnibox, tabs) is always in+  the tree, but `chrome://` **page** controls (a card's "Remove") may be **absent on a cold page** —+  Chromium builds the content tree only once it detects an assistive-tech client. **Pattern:**+  `find_control` → if found+invokable, `ui_click`; if `uia_not_found`/only frame controls, **retry once**+  (the query usually wakes the tree); still nothing → SendInput fallback.+- `role ∈ button edit link checkbox menuitem tab listitem combobox radiobutton document group spinner`.+  **Prefer `automationId`** when present — stable + locale-independent.++## Ghost cursor+`cursor:true` + `intent:"…"` on a click verb glides the teal Adom cursor (a trust tell). As of AD 1.8.175+it **auto-hides after 60s** (refreshed on each show/move). Extend with `ttlMs`, or `persist:true` then+hide yourself. **No need to hide "to be safe"** — only hide early to clear it sooner.++## Recipe — open a `chrome://`/`edge://` page (CDP can't)+`Ctrl+N` (a NEW window so a background tab elsewhere can't steal the active tab) → `Ctrl+L` → type the URL+→ `Enter`, scoped to the new window's `hwnd`:+```+desktop_press_key {hwnd:<NW>, keys:["ctrl+n"]}      # then find the new window's hwnd+desktop_press_key {hwnd:<NW>, keys:["ctrl+l"]}+desktop_type      {hwnd:<NW>, text:"edge://extensions"}+desktop_press_key {hwnd:<NW>, keys:["enter"]}+```++## Recipe — UNINSTALL the extension (proven; simplified for 1.8.175)+1. Open `edge://extensions` (recipe above).+2. `find_control {hwnd:<NW>, role:"button", contains:"Remove"}`. `invokable` → `ui_click` it (background,+   no bubble-focus dance). Page tree cold → retry once → else `desktop_click {x:rect.centerX,+   y:rect.centerY}`.+3. The **"Remove '<name>'?" confirm is a borderless popup bubble** anchored to the toolbar icon —+   invisible to `screenshot_window`, absent from `list_windows`. So: **`desktop_screenshot_screen`** → SEE+   the bubble → **`desktop_click {space:"image", shotId, x, y}`** on its "Remove".+   *(Pre-1.8.175 hack, no longer needed: open the bubble, then `desktop_press_key {keys:["enter"]}` with+   NO `hwnd` so focus isn't stolen — the bubble auto-dismisses on focus loss.)*+4. **Verify by state, not pixels:** poll **`nbrowser_profiles`** — the `edge:`/`chrome:` key **drops** when+   the SW dies. (The page render lagged/looked unchanged and burned an hour the first time.)++## Recipe — REINSTALL (Load unpacked)+1. Click **"Load unpacked"** (find_control → `ui_click`, or image-space click) → opens a Win32 **folder+   picker** (`#32770`; it DOES enumerate and has an a11y tree).+2. Put the extension dir in the **Folder:** field, then **Select Folder**:+   `ui_set` the Folder edit (or `desktop_type` the path, e.g.+   `C:\Users\<user>\adom-browser-extension\extension`) → `ui_click "Select Folder"`.+3. Load-unpacked reads the **on-disk** build (version may jump). Verify the `edge:`/`chrome:` key returns+   to `nbrowser_profiles` (reconnects in ~1–2s).++## Why these traps exist (so you don't re-discover them)+- **"My click did nothing"** was an illusion — the click worked; the confirm bubble was just *invisible to+  window-capture*. Use `screenshot_screen`.+- **SendInput foregrounds** (it lands only on the foreground window) → interrupts the user *and* dismisses+  focus-sensitive bubbles. Prefer **UIA `ui_click`**; if you must SendInput to confirm a focus-sensitive+  popup, **don't pass an `hwnd`** (so AD doesn't raise/steal focus).+- **Don't pixel-hunt.** Image-space click (`space:"image"`) or `find_control` rect. Don't fetch the window+  rect to compute scale — `coordMap` already has it.++Canonical `desktop_*` verb reference: https://wiki.adom.inc/adom/adom-desktop