Hydrogen Desktop — Screen Capture & Sharing (the AI-consent surface)

This is the surface you (the AI) trigger to see HD's own screen. To screenshot the editor (panels, the workspace, or the whole monitor) into the container, a screen share must be active first. You request it, the user consents once (HD auto-approves in ~3s), a brief countdown plays, then you can capture frames.

This is not the same as host OS screenshots. desktop_screenshot_* (see hd-adom-desktop) take true OS screenshots via AD with zero dialogs. This skill is the in-webview share that captures what HD itself is rendering (the editor). Inside HD, frames are grabbed natively via CDP — there is no getDisplayMedia; the browser getDisplayMedia/Element-Capture path applies only to web-Hydrogen. Either way the share is gated by an in-app consent dialog. And it is distinct from hd-permissions (WebView2 auto-granting mic/cam OS permissions) — that's a different layer.

The Screen Capture dropdown (editor toolbar)

Source: src/lib/components/editor/EditorNav.svelte (~L2227–2332). Store: src/lib/stores/screenCaptureStore.ts.

When not yet sharing, the dropdown offers two ways to start:

Option Description What you get
Share this tab "Panels and workspace screenshots" Captures HD's own webview — panels + workspace
Share entire screen "Full display including other apps" Captures the whole monitor (or a chosen window)

While sharing, it shows Sharing <surface> ("entire screen" / "window" / "this tab") plus capture actions:

Action Description
Workspace screenshot Save to container's screenshots folder
Full screen screenshot Save to container's screenshots folder (only when the surface is a monitor/window)
Stop sharing Ends the share

Window mode toggle (Webview vs Pup)

Below the share options is a Window mode toggle with two buttons — Webview and Pup (windowModeStore). It selects which window surface capture/recording targets: HD's own embedded Webview, or a Pup (Puppeteer) browser window. Webview = "see HD itself"; Pup = "see the AI-driven browser."

Capture scopes & how a frame is taken

Inside HD, capture is native (CDP), not getDisplayMedia. Tab/screen snapshots come from the native CDP path — no live MediaStream, no browser screen-picker. The store still flips isSharing and sets displaySurface so the same UI/consent flow applies. The getDisplayMedia/Element-Capture machinery described below is the web-Hydrogen-only path; HD reuses its scopes and consent surface, but grabs the actual pixels via CDP.

screenCaptureStore.ts exposes the scope type and the capture functions:

type CaptureScope = 'screen' | 'workspace' | 'panel';
Scope Function What it captures
panel capturePanel(panelId) a single editor panel
workspace captureWorkspace() the whole HD workspace surface
screen captureScreen() the full display / shared window

Other store entry points you'll see: startTabCapture(), startScreenCapture(), stopScreenCapture(), captureFrame(), isCaptureActive(), promptScreenCapture(message?) (asks the user to enable sharing).

State shape: { isSharing, displaySurface, hasElementCapture, promptMessage }, where displaySurface is 'browser' (tab) / 'window' / 'monitor'.

Element Capture (RestrictionTarget) — web-Hydrogen only

This path only runs in web-Hydrogen (not inside HD, which captures via CDP — see the note above). When 'RestrictionTarget' in window and getDisplayMedia are available, it can restrict a tab capture to a single DOM element so only that panel is in frame: restrictCaptureTo(el) builds a RestrictionTarget.fromElement(el) and calls track.restrictTo(target) (a short ~3s settle window). That is how a panel-scoped capture yields just the panel, not the whole webview. Inside HD, the same panel scoping is achieved through the native CDP capture instead.

The AI sharing-approval flow

When you (or a bridge) need screen/mic access, HD shows a consent gate before any capture. Source: src/lib/components/editor/SharingApprovalDialog.svelte, store src/lib/stores/sharingApprovalStore.ts.

The request you send carries:

{ bridgeKey, reason, shareType: 'tab' | 'screen' | null, audio: boolean }
  • The dialog header reads "AI wants access" and shows your reason in a highlighted box. Send a clear, specific reason — it's the only thing the user sees to decide. ("Screenshot the schematic panel to verify the layout," not "capture screen.")
  • HD auto-approves after ~3 s (AUTO_MS = 3000), showing "Auto permission will be given in N…". This is the HD-only fast path so you're not blocked on hands-free flows.
  • The user can override before it auto-fires: flip share-type between Share this tab and Share entire screen, toggle the mic, or pick a mic device. Any interaction cancels the auto-countdown so they stay in control.
  • On approve, HD starts the screen share first (tab or screen), then mic if requested, and returns a result (status: 'approved', with sharing + mic details) to the requesting bridge.

CountdownToast — the cancelable lead-in

Source: src/lib/components/editor/CountdownToast.svelte. Before capture/recording actually starts, a countdown toast ("Recording in N…", default 3) plays, optionally showing the sources and reason. It has a cancel (×) — the user can abort right up to the last second. Don't assume capture started the instant you requested it; the toast is a real, abortable delay.

What this means for you (the AI)

  1. To screenshot HD's editor you need an active share. If you're not sharing, call promptScreenCapture(...) / send a sharing request with a reason; don't silently fail a capturePanel/captureWorkspace.
  2. Write the reason for a human. It's your one chance to justify the access.
  3. Respect the override + countdown. The user may switch share-type, deny mic, or cancel the countdown — read the returned result before claiming success.
  4. Pick the right scope: panel (one panel, Element-Capture-restricted), workspace (the HD surface), or screen (full display). Use Window mode (Webview vs Pup) to choose HD's own UI vs the AI browser.
  5. For host-OS screenshots that need no share at all, use AD's desktop_screenshot_* instead (see Related skills).
  • hd-recording — screen/mic recording (same share + countdown machinery; in-app .webm recorder + AD desktop recording)
  • hd-self-screenshot — CDP per-panel screenshots + shotlog visual-verify loop
  • hd-permissions — WebView2 auto-granting OS mic/cam/notification permissions (a different layer from this consent dialog)
  • hd-adom-desktop — AD's dialog-free host OS screenshots (desktop_screenshot_screen/_window)