← Commit history

Publish 1.1.0

John Lauer ·8c8ab942f1 ·3mo ago ·parent 6a1bcca
5 files changed +92−60
README.md+3−2
@@ -48,8 +48,9 @@ adompkg install adom/adom-desktop-control - **hydrogen-web-control** - what's reachable in the Hydrogen editor and why its embedded   Claude input is not. - **claude-control** - when to delegate to a sibling Claude vs do it yourself.-- **screen-recording** - `desktop_record` + `desktop_caption`, with the foreground-rebump,-  recorder-reset, and caption-hygiene rules that make it actually work.+- **screen-recording** - WGC single-window capture (`desktop_record_window_start`):+  background, no picker, no banner, occlusion-safe, the right way to record a window+  (whole-desktop capture and its foreground theft is the wrong default). - **video-post-production** - crop, speed up, brand-font captions, Ken Burns on stills,   TTS voiceover, concat, and wiki-ready keyframes. 
SKILL.md+8−4
@@ -28,7 +28,7 @@ pack of tested patterns, each one distilled from a real session, each labeling | A native app window (Fusion, dialogs, non-web UI) | `desktop_*` (UIA + SendInput) | [desktop-ui-control](skills/desktop-ui-control/SKILL.md) | | The Hydrogen web editor / its Claude panel | (mostly can't, read why first) | [hydrogen-web-control](skills/hydrogen-web-control/SKILL.md) | | A sibling Claude tab to run a task | depends on where it runs | [claude-control](skills/claude-control/SKILL.md) |-| Capture the screen to a video | `desktop_record_*` + `desktop_caption` | [screen-recording](skills/screen-recording/SKILL.md) |+| Capture a window to a video (the right way) | `desktop_record_window_start` (WGC, background) | [screen-recording](skills/screen-recording/SKILL.md) | | Turn raw clips into a finished demo | ffmpeg + `video-post` | [video-post-production](skills/video-post-production/SKILL.md) |  ## The cross-cutting rules (true on every surface)@@ -44,11 +44,15 @@ pack of tested patterns, each one distilled from a real session, each labeling - **Upload files via DataTransfer, not the OS file picker.** Inject the bytes into the   page and set the `<input type=file>`, works in pup and nbrowser. The OS picker is a   last resort (needs foreground SendInput). See native-browser-control / pup-browser-control.-- **Recording captures whatever is topmost.** If you drive via background CDP while-  recording, the recording shows the WRONG window unless you re-foreground the target-  before every visible step (the "foreground re-bump"). See screen-recording.+- **To record, capture the WINDOW (WGC), not the whole desktop.** `desktop_record_window_start`+  records one window in the background even when it's occluded, so there is no foreground+  theft. Whole-desktop capture (`desktop_record_start`) records whatever is topmost, so a+  self-updating window (e.g. the Hydrogen panel) keeps stealing the take. See screen-recording. - **Verify by screenshot, don't assume.** After any click/type into a webview, screenshot   and confirm the text/state actually landed, OS input silently no-ops on many webview editors.+- **Search the verb surface before you assume a feature is missing.** `adom-desktop help+  <namespace>` / `<verb>`. Several capabilities (WGC window recording, nbrowser on Edge,+  nbrowser recording) were missed by assuming they didn't exist. Look first.  ## Install 
package.json+1−1
@@ -4,7 +4,7 @@   "type": "skill",   "title": "Adom Desktop Control",   "brief": "Drive the user's real desktop from the cloud: their Chrome/Edge, headless pup, native UI, the Hydrogen editor, sibling Claude tabs, plus screen recording and video post.",-  "version": "1.0.1",+  "version": "1.1.0",   "description": "A skill pack of hard-won, tested patterns for driving an Adom user's actual machine from the cloud AI through Adom Desktop: their real logged-in Chrome/Edge via the browser extension (CDP), headless pup browsers, native UI via UIA and SendInput, the Hydrogen web editor, sibling Claude tabs, and a full screen-recording + video post-production pipeline. Every skill is distilled from real sessions and labels what works, what fails, and why.",   "org": "adom",   "dependencies": {},
page.json+1−1
@@ -4,7 +4,7 @@   "type": "skill",   "title": "Adom Desktop Control",   "brief": "Drive the user's real desktop from the cloud: their Chrome/Edge, headless pup, native UI, the Hydrogen editor, sibling Claude tabs, plus screen recording and video post.",-  "version": "1.0.1",+  "version": "1.1.0",   "description": "A skill pack of hard-won, tested patterns for driving an Adom user's actual machine from the cloud AI through Adom Desktop: their real logged-in Chrome/Edge via the browser extension (CDP), headless pup browsers, native UI via UIA and SendInput, the Hydrogen web editor, sibling Claude tabs, and a full screen-recording + video post-production pipeline. Every skill is distilled from real sessions and labels what works, what fails, and why.",   "org": "adom",   "dependencies": {},
skills/screen-recording/SKILL.md+79−52
@@ -1,69 +1,96 @@ --- name: screen-recording description: >-  Record the user's desktop to a video clip via Adom Desktop, with large on-screen-  captions, and the gotchas that make or break it (foreground re-bump, recorder-  reset, pulling the file). Trigger words: record the screen, screen recording,-  desktop_record, record a demo, capture the screen, on-screen captions,-  desktop_caption, record the browser flow.+  Record the user's screen to a video clip via Adom Desktop. THE right tool is+  WGC single-window capture (background, no picker, no banner, occlusion-safe), not+  whole-desktop capture. Covers the full recording verb surface, on-screen captions,+  and the gotchas that make or break it. Trigger words: record the screen, screen+  recording, desktop_record, record a window, WGC, record a demo, capture the screen,+  on-screen captions, desktop_caption, record the browser flow, record in the background. --- -# Screen recording (desktop capture + captions)+# Screen recording -Capture the whole desktop to a `.webm` while you drive an app, with big AD-rendered-captions burned into the capture. Hard-won, every rule below comes from a failure.+> **Search the verb surface before you assume.** This skill exists because an agent+> assumed whole-desktop capture was the only option and that nbrowser had no recording,+> both wrong. Run `adom-desktop help record` (the `desktop_recording` namespace, 10+> verbs) and `adom-desktop help <verb>` FIRST. The right tool was hiding in plain sight. -## The verbs+## Use WGC single-window capture (the right tool, almost always)++`desktop_record_window_start` records ONE window via **Windows Graphics Capture (WGC) ++Media Foundation**. This is what you want for recording a browser flow or an app:++- **No picker, no capture banner** (border suppressed on Win11).+- **Background-capturable**: the target window can be **occluded or off-screen** and it+  still records. So you do NOT need to foreground anything, the foreground-theft problem+  (another window stealing focus) simply does not apply.+- Resolve the window by `hwnd` (from `desktop_list_windows` / `desktop_find_window`) or+  `titleContains`, same as `desktop_screenshot_window`.  ```bash-# whole-desktop capture (getDisplayMedia, 30fps). recordingId is at the TOP LEVEL of the response.-adom-desktop desktop_record_start '{"confirmDesktopNotTabRecording":true,"reason":"<why>","fps":30,"audio":false,"monitor":"primary"}'-adom-desktop desktop_record_stop  '{"recordingId":"rec-N"}'        # -> filePath, durationMs, sizeKB (top level)-adom-desktop desktop_recorder_close '{}'                          # reset stale recorder state (see below)--# large always-on-top click-through caption, captured by the recording:-adom-desktop desktop_caption '{"text":"...","size":"large","position":"bottom","id":"d","persist":true}'-adom-desktop desktop_caption '{"action":"force-clear"}'           # nuke ALL captions+# start (recordingId is nested under output.data; path is top-level)+adom-desktop desktop_record_window_start '{"hwnd":17830478,"fps":30,"codec":"h264"}'+# ... drive the page via nbrowser_eval / browser_eval (also background) ...+adom-desktop desktop_record_window_stop '{"recordingId":"rec-<hex>"}'   # -> .mp4 path+# the .mp4 lands under %TEMP%\adom-desktop-recordings\ ; pull_file it (lands in /tmp here) ``` -For a single browser tab you can also `browser_record_*` (pup), but it's low-fps CDP-screencast, prefer desktop capture for quality. See-[pup-browser-control](../pup-browser-control/SKILL.md).--## The five rules that make it work--1. **Parse `recordingId` from the TOP LEVEL** of `desktop_record_start`'s response, NOT-   from a nested `output` field. Getting this wrong = empty RID = no recording.-2. **Reset the recorder first.** It goes stale (accumulated clips) and silently returns-   no recordingId / a 439-byte empty file. Call `desktop_recorder_close` + `sleep 2`-   before `desktop_record_start`. Run it inline / via the background tool, NOT detached-   with `&` (detached runs produced empty files).-3. **Foreground re-bump before EVERY visible step.** The capture shows whatever is-   topmost. If you drive via background CDP, the recording shows the WRONG window unless-   you `desktop_bring_to_front` the target before each click/scroll. This was THE fix for-   "my recording captured the wrong app." Re-bump, then act.-4. **Force-clear captions before each recording.** `persist:true` captions survive across-   recordings and leak a stale caption into the next clip. `desktop_caption {action:"force-clear"}`-   first. (Or caption in post instead, see below.)-5. **Live captions drift on autonomous flows.** If the page advances faster than your-   sleeps, the live caption lands on the wrong screen. For anything whose timing you don't-   control tightly, record CLEAN (no live captions) and add them in post, see-   [video-post-production](../video-post-production/SKILL.md).--## Pull the file to post-produce+Parse: `recordingId = output.data.recordingId`, `path = top-level .path`.++### The one rule: WGC captures the window's ACTIVE tab++WGC records the window's pixels, i.e. its currently-visible tab. So **the tab you drive+via CDP must be the active tab in the recorded window.** nbrowser has no "activate tab"+verb, so: open a fresh tab with `nbrowser_open_tab` (it becomes active), confirm with+`desktop_find_window` that the window's title is your target, and drive THAT tab. Then+WGC + background CDP eval = a clean recording with zero foregrounding. Output is full-res+window-only (no desktop, no taskbar to crop).++## Whole-desktop capture (only when you truly need the whole screen)++`desktop_record_start` records EVERY pixel via Chrome getDisplayMedia. Avoid it for+single-window jobs, it has three problems WGC doesn't:++- **Foreground theft.** It captures whatever is topmost. If you drive via background CDP,+  the recording shows the WRONG window unless you `desktop_bring_to_front` the target+  before EVERY visible step (the "foreground re-bump"). A window that updates on its own+  (e.g. the Hydrogen panel showing your conversation) keeps stealing the foreground and+  ruins the take.+- **Picker + banner + wake-lock.**+- **Privacy.** It records the user's other tabs/apps.+- Gotchas if you must use it: `recordingId` is **top-level** in the response (not under+  `output`); reset stale state with `desktop_recorder_close` + `sleep 2` first (it+  silently returns no id / a ~439-byte empty file otherwise); run it via the background+  tool, NOT detached with `&` (detached runs produced empty files).++## nbrowser_record (getDisplayMedia, has a picker, inferior to WGC)++`nbrowser_record_start` exists but uses **getDisplayMedia**, so it parks in+`state:"awaiting_pick"` with a **share picker up on the user's screen**. You cannot stop+an `awaiting_pick` record via the API (`nbrowser_record_stop` errors "not recording"),+and the picker is browser chrome (not page DOM, not UIA-reachable). To clear a stuck+picker: click its **Cancel** button via a foregrounded `desktop_click` at the mapped+coordinate, or navigate the tab (cancels the pending request). **Just use WGC instead.**++## On-screen captions  ```bash-adom-desktop pull_file '{"filePaths":["C:/Users/<u>/AppData/Local/Adom Desktop/plugins/puppeteer/recordings/rec-desktop-<ts>.webm"]}'-# NOTE: it lands in /tmp on the CLI host (your container), NOT your dest arg. Find it there.+adom-desktop desktop_caption '{"text":"Every part matched","size":"large","position":"bottom","id":"d","persist":true}'+adom-desktop desktop_caption '{"action":"force-clear"}'   # nuke ALL captions ``` -## Privacy--Whole-desktop capture records EVERYTHING on the primary monitor (the user's other tabs/-apps). Maximize the target app so it dominates, and crop the taskbar / private tab strip-in post. The user must have authorized the recording (e.g. "I'm at the gym, go").+- A large always-on-top click-through overlay, captured by the recording. BUT WGC records+  a single WINDOW; a desktop caption overlay is a SEPARATE window, so WGC of the target+  window will NOT include it. For WGC clips, **caption in post** (see+  [video-post-production](../video-post-production/SKILL.md)). Use `desktop_caption` for+  whole-desktop recordings.+- `force-clear` before any recording: `persist:true` captions survive across clips and+  leak a stale caption into the next one.+- Live captions drift on autonomous flows (the page advances faster than your sleeps).+  Record CLEAN, caption in post, where you control the exact timing. -## Reliability note+## Always verify -The recorder is flaky. Verify after stop (`durationMs`, `sizeKB`), pull, and extract a-frame to CONFIRM the right window was captured before you build on it. Don't assume.+The recorder can be flaky. After stop, pull the file and **extract a frame to CONFIRM the+right window/tab was captured** before you build on it. Don't assume.