← Commit history

Install run learnings: #24 navigate now focus-neutral (verified z:1 not z:0); #23 clipboard verbs native; picker auto-fill via SendInput is UNSAFE while user active (keys hit wrong window) - clipboard-assist (user pastes) is the method; Load unpacked can open multiple pickers; cancel via contains:Cancel not automationId 2

John Lauer ·f5a6d6e5ff ·4mo ago ·parent 02f4d59
2 files changed +38−41
docs/INSTALL-FLOW.md+2−2
@@ -128,8 +128,8 @@ loaded: sometimes does not commit: below, the URL bar reads `chrome://extensions` but the page never changed (it still shows the prior content), so there is no Load unpacked button yet. Screenshot-verify the real Extensions page rendered (tab title becomes "Extensions"); re-issue `desktop_navigate` if it did not.-Note: `desktop_navigate` can also surface the window to the front to commit, so check it did not steal-focus.+(As of AD 1.9.34+, `desktop_navigate` is focus-neutral - it restores your foreground window and returns+an honest `navigated` signal - so this surfaced-the-window failure is fixed; older AD foregrounded.)  ![Navigate did not commit: the omnibox shows chrome://extensions but the page never changed](/blob/app/adom-browser-extension/screenshots/install-navigate-not-committed.png) 
skills/installing-the-extension/SKILL.md+36−39
@@ -70,13 +70,12 @@ Rules, in order:   the user to pause typing for a moment, do the single action, and return immediately. Never hold the   foreground, never do it silently, and never do it while the user is actively typing if there is any   other way.-- ⚠️ **Foreground can happen even when you NEVER call `bring_to_front`.** Opening a new window, and-  especially **`desktop_navigate`** (its omnibox commit can surface/activate the window), may bring a-  window to the FRONT despite `focused:false`. Judge by the **outcome**: after an open or a navigate,-  check `desktop_list_windows` - a window at `z:0` is topmost = you surfaced it. (Observed live here: a-  `desktop_navigate` to `chrome://extensions` left the window at `z:0` with no `bring_to_front` called.)-  Also screenshot-verify the page **actually loaded** (tab title becomes "Extensions"), not just that the-  omnibox shows the URL - the posted-Enter sometimes does not commit, and the navigate may need a retry.+- ✅ **`desktop_navigate` is now focus-neutral (AD 1.9.34+).** It restores your prior foreground window+  after committing and returns an honest `navigated` signal (+ `titleChanged`). Verified live: a+  `chrome://extensions` navigate committed (`navigated:true`, tab title "Extensions") while my window+  stayed at `z:1` and the user's window kept `z:0`. (On OLDER AD it foregrounded - the general principle+  still holds: judge foreground by the **outcome**, `desktop_list_windows` `z:0` = topmost = you surfaced+  it.) Still screenshot-verify the page actually rendered, not just the `navigated` flag.  ## The flow (background; never foreground the user's work) Open a fresh background window you own, get its `hwnd`, and drive **that** (never the user's tabs):@@ -108,41 +107,39 @@ desktop_ui_click {hwnd:<W>, automationId:"loadUnpacked"}          # VERIFIED sta ``` The picker is a Win32 dialog (`#32770`, title "Select the extension directory."). ⚠️ **It opens in an arbitrary last-used directory (observed: Downloads) - NEVER trust "it remembers the folder."** You MUST-set the path. ⛔ The whole catch: filling this field needs keyboard input, and SendInput-(`desktop_type`/`desktop_press_key`) **foregrounds the window** - the rude, work-corrupting move above.-So prefer the path that keeps the user in control:+set the path. -1. **Clipboard assist - PREFERRED when the user is at the machine (you never steal focus).** Put the-   absolute path on the clipboard, `desktop_caption {text:"Adom: paste this path (Ctrl+V) into the Folder-   box, then click Select Folder"}`, and let the USER do the single paste + click. You stay background;-   they finish the one keyboard step. Confirm by screenshot that the card loaded.-   - ⚠️ **AD has no clipboard verb yet**, so set the clipboard via `shell_execute`:-     `powershell -NoProfile -Command "Set-Clipboard -Value '<abs path>'"` (read back with `Get-Clipboard`).-     **Be polite: save the user's current clipboard first (`Get-Clipboard`) and restore it after**, since-     you are overwriting it. (A native `desktop_clipboard_set`/`desktop_clipboard_get` would remove the-     quoting hassle - tracked as an AD request.)-2. **Address bar via SendInput - VERIFIED to work, but it FOREGROUNDS (last resort; user away/consenting).**-   First `desktop_caption {text:"Adom: one moment, please don't type"}`, then:-   ```-   desktop_press_key {hwnd:<picker>, keys:"ctrl+l"}             # focus the picker address bar (FOREGROUNDS)-   desktop_type      {hwnd:<picker>, text:"<abs path to extension>"}-   desktop_press_key {hwnd:<picker>, keys:"enter"}             # navigates INTO the folder (breadcrumb confirms it)-   desktop_ui_click  {hwnd:<picker>, automationId:"1"}         # "Select Folder" (id "1", invokable) -> commits-   ```-   Do this ONLY when the user isn't typing; keep it to this one quick burst, then leave focus alone.-3. **UIA SetValue does NOT work here (verified).** The Folder combobox has **no ValuePattern**-   (`desktop_ui_set` -> "control found but has NO ValuePattern"). There is no clean background fill for-   this field - which is exactly why clipboard-assist (1) beats foregrounding.-4. **Coordinate click + type (most fragile + foregrounds).** `desktop_click` the Folder box by its rect,-   then `desktop_type`. Layout-dependent; avoid.+> ⛔⛔ **HARD-LEARNED (live, 2026-06-28): do NOT auto-fill the picker with SendInput while the user is at+> the machine.** The picker opens **BEHIND the user's active windows** (observed `z:1`/`z:2`, never `z:0`;+> the foreground was the user's VS Code / app). `desktop_press_key`/`desktop_type` into it did **not**+> fill the picker and the keystrokes most likely went to the **user's foreground window instead** (risking+> pasting your path / a stray Enter into their editor). UIA `desktop_ui_set` also can't fill it (the Folder+> combobox has **no ValuePattern**). So there is **no safe, reliable background way for the agent to type+> the path.** Use clipboard-assist: -**ALWAYS screenshot and READ the path BEFORE committing.** `desktop_screenshot_window {hwnd:<W>}` (the-BROWSER hwnd) returns the picker in its `screenshots[]` array - confirm the breadcrumb/Folder shows-`...\extension` before clicking **Select Folder**; the wrong folder loads a broken/old build silently.-To back out without loading: `desktop_ui_click {hwnd:<picker>, automationId:"2"}` (Cancel).+**Clipboard-assist (the method - the USER does the one keyboard action):**+1. Stage the path on the clipboard (native verb, background, no focus steal):+   `desktop_clipboard_set {text:"<abs path to extension>", returnPrior:true}` (AD v1.9.35+). Keep the+   returned `prior` and **restore it afterward** with another `desktop_clipboard_set`. ⚠️ The clipboard is+   **volatile** (other apps/the user overwrite it) - set it **immediately** before asking the user, and+   `desktop_clipboard_get` to confirm it still holds the path.+2. `desktop_caption {text:"Adom: the folder picker is open - paste the path (Ctrl+V) into the Folder box, then click Select Folder"}`.+3. The user pastes + clicks Select Folder. Then `desktop_screenshot_window` to confirm the card loaded. -> Next: a deterministic `install/load-unpacked` helper that runs ladder step 1 with retries + the-> screenshot path-check + captions baked in, so this is one call, not improvised steps.+**Agent-driven fill is ONLY acceptable when the user is truly away** (and even then it is finicky): first+`desktop_bring_to_front {picker hwnd}` so the picker is genuinely `z:0` (verify with `desktop_list_windows`),+caption a "don't type" warning, then `ctrl+l` -> `desktop_type <path>` -> `enter` -> screenshot-verify the+breadcrumb shows `...\extension` -> click Select Folder. Never do this blind while they are working - the+keys will not reliably reach the picker.++**Gotchas observed live:**+- **Load unpacked can open MULTIPLE pickers** (saw 2 from one flow). Before filling, `desktop_list_windows`+  and ensure exactly one `#32770` "Select the extension directory." is open; cancel extras.+- **Cancel the picker with `desktop_ui_click {hwnd:<picker>, contains:"Cancel"}`** - `automationId:"2"`+  returned success but did **not** close it; the name-based click did. ("Select Folder" likewise: prefer+  `contains:"Select Folder"` over `automationId:"1"`.)+- **ALWAYS screenshot + READ the path BEFORE committing** (`desktop_screenshot_window {hwnd:<browser>}`+  returns the picker in `screenshots[]`); the wrong folder loads a broken/old build silently.  ### Step C - verify it loaded clean AND connected (screenshot FIRST, three checks) 1. **Screenshot the extensions list and READ it** - this is the check that actually matters: