---
name: desktop-ui-control
description: >
  Click, type, and read NATIVE app windows (Fusion, dialogs, non-web UI) on the
  user's desktop via Adom Desktop UIA and SendInput. Covers find_control/ui_click,
  click/type/press_key, screenshot-to-coordinate mapping, and window foreground.
  Trigger words: UIA, desktop_click, desktop_type, click a native window, drive a
  dialog, find_control, press_key, bring window to front, coordinate click.
---

# Desktop UI control (native windows)

For NON-web app windows (Fusion, OS dialogs, OAuth account choosers rendered outside
a reachable page), use the `desktop_*` verbs. Two mechanisms, pick by what works:

## 1. UIA, programmatic, background (prefer this)

```bash
adom-desktop desktop_list_windows '{}'
adom-desktop desktop_find_window '{"titleContains":"Fusion"}'           # -> hwnd, rect
adom-desktop desktop_find_control '{"hwnd":<h>,"nameContains":"Continue"}'  # -> best.{invokable,settable,rect}
adom-desktop desktop_ui_click '{"hwnd":<h>,"nameContains":"Continue"}'  # InvokePattern, NO foreground/focus steal
adom-desktop desktop_ui_set   '{"hwnd":<h>,"nameContains":"Name","value":"X"}'  # SetValue
adom-desktop desktop_screenshot_window '{"hwnd":<h>}'                    # captures a window even in the BACKGROUND
```

`desktop_find_control` returns `best.invokable` / `best.settable`. If `invokable:true`,
`desktop_ui_click` works in the background (Invoke/SetValue are programmatic, no cursor
move, no focus steal). **Chromium/Edge expose their a11y tree, so page buttons are
reachable by accessible name**, but NOT shadow-DOM or webview-internal controls.

## 2. SendInput, foreground only (fallback)

When a control isn't in the UIA tree (shadow DOM, canvas, OAuth pages that block eval):

```bash
adom-desktop desktop_bring_to_front '{"hwnd":<h>}'        # SetForegroundWindow
adom-desktop desktop_click '{"x":1622,"y":671}'          # SendInput at SCREEN pixels
adom-desktop desktop_type  '{"text":"hello","hwnd":<h>}' # WM_CHAR unicode into the focused control
adom-desktop desktop_press_key '{"hwnd":<h>,"keys":["ctrl","l"]}'  # chords + named keys
```

### Screenshot -> screen-coordinate mapping (for desktop_click)

`desktop_screenshot_window` returns a scaled image (e.g. 1500x900) of a window whose
real rect (from `desktop_find_window`) is e.g. 2560x1528 at left=2,top=0. To click an
element you see at image `(ix,iy)`:

```
scale = rect.width / image.width
screen_x = rect.left + ix*scale
screen_y = rect.top  + iy*scale
```

Overlay a coordinate grid on the screenshot to read `(ix,iy)` precisely, then click.
Proven driving a Google OAuth account chooser this way.

## What SendInput CANNOT do (hard-won)

- **OS input no-ops on many webview editors.** SendInput reaches a browser's NATIVE UI
  (address bar, search, Edge's command palette) but **does not register** in some webview
  text editors (Hydrogen's Claude input, VS-Code-in-browser editors). `desktop_type`,
  `desktop_press_key`, and click+type all silently fail there. ALWAYS screenshot to verify
  the text landed; if it didn't, the editor only accepts CDP-injected input.
- See [hydrogen-web-control](../hydrogen-web-control/SKILL.md) for that specific dead end.

## Foreground hygiene

`desktop_bring_to_front` / SetForegroundWindow steals focus, the user notices. Use it
ONLY for SendInput, never for UIA or screenshots (both work in the background). Don't
foreground a window the user might be using. The exception is RECORDING, where you must
re-foreground the target before each step (see [screen-recording](../screen-recording/SKILL.md)).