# pup Interactive Surface: toolbar extension, drawing cursors, snip return path, move-to-webview

**Status:** phases 1 + 2 SHIPPED 2026-08-10 (pup 2.0.50 to 2.0.58: cursors, tight-crop, quick
capture, clipboard rail, toolbar + identity extensions on CfT, view-toggle symmetry) · phase 3
move-to-webview DEFERRED as a future TODO (John 2026-08-10: not enough value right now) —
tracked in the pup-bridge issue filed the same day · container push also deferred (TODO in
phase 2 section)

## Context

Annotate + snip shipped in pup 2.0.49 (overlay + jump-list task; capture stays in-DOM so the
owning thread's next screenshot shows it). John's follow-ups define a coherent feature set:
actions must be discoverable IN the browser (not just jump lists), drawing needs real tool
cursors, a Done'd snip must reach the owning AI thread's chat, and a pup window should be able to
move back to a webview in the workspace it came from. Exploration (3 agents) resolved the
architecture questions; this plan is the build order.

**Headline decisions**
- **Toolbar lives in Chrome's TOP BAR via a small extension** (John: "menu item in the chrome bar
  up at the top rather than an overlay in their page content... like Claude Desktop's"). Browser
  chrome is unreachable from CDP, so an MV3 extension action icon is the only way to put a button
  there — and it's the RIGHT shape: the extension is a thin trigger surface; pup stays the engine
  (extension popups may fetch pup's loopback `/command` with host_permissions — the mixed-content
  wall only blocks web pages). The in-page pill is DROPPED.
- **Return path = CLIPBOARD-FIRST** (John's pick): on mouse-up the tight-cropped PNG lands in
  the Windows clipboard as image + text/plain JSON sidecar together (multi-format DataObject), so
  one paste into any AI chat carries both; the PNG also bakes a caption bar + iTXt metadata so an
  image-only paste still carries context. Push-back-to-container via ab: DEFERRED (TODO, ab
  prompt ready in phase 2).
- **Move-to-webview: build the local Hydrogen (ah) leg NOW** (zero new ab verbs needed); hc leg
  is blocked on one cloud-side auth decision (filed as an ask).

## Phase 1 — pup 2.0.50: drawing cursors, quick capture + the toolbar extension

All in `/home/adom/project/adom-desktop-puppeteer-bridge/src/server.js`.

**1a. Photoshop-style tool cursors** (in `ANNOTATE_OVERLAY_JS`, ~L5260):
- Inline-SVG data-URI cursors, hotspot at the working tip: pen → pencil glyph with a live swatch
  of the current color (cursor rebuilt on color change); arrow → fletched pointer; box + snip →
  precise crosshair; text → I-beam. No external assets (CSP-safe).

**1a2. Tight-crop annotation captures (John's ask — beat Claude Desktop on tokens)**:
- On Done, compute the bounding box of ALL drawn strokes (the overlay already holds every
  stroke's points), pad by a ~48px margin, clamp to the viewport, and capture ONLY that clip —
  not the whole browser view ("if the user wants the whole view, they'll draw a circle around
  it"). Text notes count toward the box (measure the rendered text width). Snip is unchanged
  (already exact-region). The full-view drawing still stays in-DOM for the owning thread's
  next full screenshot, so no information is lost — the CAPTURED PNG is just the tight,
  token-cheap version that rides the return path.

**1a3. Quick capture on mouse-up (John's ask)** — an "⚡ Auto-send" toggle in the overlay toolbar
(also the default mode when launched via ▣ Snip or the hotkey):
- **The moment the mouse button lifts**: capture fires immediately (tight-crop rules from 1a2) —
  no Done click. If the user keeps drawing, each subsequent mouse-up recaptures with the union
  bounding box and supersedes the previous capture (single circle = instant; multi-stroke = last
  lift wins with everything included; dedupe by capture id downstream so the thread sees one).
- **Clipboard**: the PNG is copied to the Windows clipboard at the same instant (via ab clipboard
  verb if present, else PowerShell `Set-Clipboard` on the saved file) — fastest path is often the
  user pasting straight into whatever chat window they're in.
- **Confirmation toast WITH preview**: instantly on mouse-up, an in-page toast (bottom-right,
  same visual family as the toolbar) shows "▣ Snippet copied to your clipboard · sent to
  <thread>" WITH a thumbnail of the ACTUAL captured PNG (data-URL preview) — the user sees their
  red circle and the margin that was grabbed, verifying the crop got what they wanted before they
  tab away. Auto-dismisses ~4s; click-to-dismiss; a "↻ recapture" link if the crop missed.
- Phase 2's clipboard write (image + sidecar) fires from this same mouse-up event.

**1b. The "Adom Pup" toolbar extension** — a button in Chrome's TOP BAR (the extension action
area right of the omnibox), Claude-Desktop-style:

```
  ← →  ⟳ │ wiki.adom.inc/adom/adom-bridge          │ ⧉ │ [⠿]  ←  Adom action icon
                                                          ╭──────────────────────────╮
                                                          │ ✎ Annotate               │
                                                          │ ▣ Snip a region          │
                                                          │ ◐ Switch to public view  │
                                                          │ ▦ Open pup dashboard     │
                                                          │ ⓘ Owned by: kicad-dev    │
                                                          │ ⇱ Move to my workspace   │  (phase 3)
                                                          │ ✕ Close this window      │
                                                          ╰──────────────────────────╯
```

- **MV3 extension, thin trigger surface only**: the popup fetches pup's loopback
  `http://127.0.0.1:<port>/command` directly (host_permissions `http://127.0.0.1/*` — extensions
  are exempt from the mixed-content wall that blocks web pages). Port discovery: probe the
  known pup port range against `/health` (`bridge:"puppeteer"` fingerprint), cache in
  chrome.storage. All engine logic stays in pup; the popup sends the SAME verbs the jump lists
  send, with caller `'user-toolbar'` (added to the new `USER_CALLERS` exemption set alongside
  `'user-taskbar-menu'`, refactoring the 4 exact-string compares at L10463/L10491/L10510/L11579).
- **Per-tab owner badge**: the action icon carries a small colored badge per tab
  (`chrome.action.setBadgeText({tabId})`) with the owning thread's initials/color, pulled from
  pup `/status` — the "whose window is this" signal right in the toolbar.
- **Keyboard command**: `Ctrl+Shift+A` → snip pre-armed (chrome.commands), the Win+Shift+S muscle
  memory John described.
- **Loading strategy (the one hard constraint, verified 2026-08-10)**: branded Chrome ≥137
  REMOVED `--load-extension`; it still works on Chrome for Testing/Chromium. The CDP replacement
  (`Extensions.loadUnpacked` / Puppeteer `browser.installExtension`) requires launching with
  `--remote-debugging-pipe`, which conflicts with pup's port-based session recovery (recovery
  reconnects via DevToolsActivePort after a bridge respawn). So:
  1. **Now (John/dev)**: pup pushes the unpacked extension dir to the desktop; ONE-TIME manual
     "Load unpacked" into the pup profile (chrome://extensions, dev mode). Persists in the
     durable adom-you profile across every launch. Dev-mode startup bubble is the known cost.
  2. **Fleet**: publish "Adom Pup" UNLISTED on the Chrome Web Store (the org already has store
     presence via nb, the native-browser extension); pup's welcome flow deep-links it — one click
     per profile, no dev-mode nag.
  3. **Later investigation**: pipe-based launch + `installExtension` for zero-click automation,
     ONLY if recovery can be made pipe-compatible (do not risk the recovery architecture for it).
- **Settings (explicit user toggle, John's ask)**: a visible row in pup's dashboard Settings
  dialog — `toolbarExtension` master toggle (default on) with sub-toggles `tbOwnerBadge` (the
  per-tab thread badge) and `tbSnipHotkey` (Ctrl+Shift+A), wired exactly like the jump-list
  family (SETTINGS_DEF row + set-setting whitelist + `_chosen` stamping + live apply: OFF stops
  the badge feed and tells the extension to disable itself via its next /status poll; ON
  re-enables without a browser restart). The annotate feature keeps its own existing `jlAnnotate`
  toggle. Jump lists stay: same actions in both places. The in-page pill is dropped.
- **First deliverable is render-only**: the popup UI built + loaded in a test profile,
  screenshot for John's sign-off before wiring actions.

## Phase 2 — pup 2.0.51: the clipboard-first return path (container push DEFERRED)

**Primary rail (John's pick): the clipboard, with a JSON sidecar that survives pasting.**
On quick-capture mouse-up (or Done):
1. Tight-crop PNG (1a2) is finalized with:
   - a **baked caption bar** (thin strip under the crop: `pup · <thread>'s window · <url> · <ts>`)
     — visible to ANY AI the image is pasted into, in any chat product; toggleable in settings;
   - an **iTXt metadata chunk** carrying the full JSON sidecar {thread, container, sessionId,
     url, ts, shapes, viewport} for tooling that reads PNG metadata (sharp, already a dep,
     handles both).
2. **Multi-format clipboard write**: image + text/plain set TOGETHER via one .NET `DataObject`
   (small PowerShell -STA helper pup spawns on the desktop; plain Set-Clipboard cannot do
   multi-format). The text/plain payload is a one-line human-readable header + the JSON sidecar.
   Pasting into Claude/Codex/Kimi chat inputs attaches the image AND inserts the text (browsers
   expose both items to the paste event); custom mime formats do NOT survive into web inputs, so
   plain text is the only sidecar format used.
3. Toast (1a3) already confirms with the preview thumbnail; add "sidecar: copied" line.
4. Local archive under `SHOTS_DIR` + `session._lastAnnotation` stays (owning thread can still
   fetch via pull_file when asked).

**DEFERRED (TODO — revisit after the clipboard rail proves out): push-back-to-container via ab.**
Too complex for now (John). When revived: ONE new ab verb over the existing relay ws —
`container_deliver {server, files, note, inbox}` — then the hook mailbox + inotify instant-push
design below. The ready-to-paste AI prompt for the ab build thread (John's workflow: paste to ab,
ab replies with a contract prompt to paste back to pup):
> pup needs a reverse-delivery verb: the user annotates/snips a page in a pup window on the
> desktop, and the captured PNG + a JSON note must reach the cloud container whose AI thread owns
> that window. You already hold the connection: each container's relay server is a ws ab dials.
> Add `container_deliver {server, files:[{path}|{b64,name}], note, inbox}`: desktop-side ab
> streams the files over that ws to the container-side relay, which writes them to
> `~/.adom/inbox/<inbox>/<ts>-<id>/` (files + note.json) and acks. Size cap ~15 MB, errors honest
> (server not connected, disk). Called by pup via the SDK callback channel with caller-identity
> headers, so log it in the activity log like any verb. When it's built and shipped, reply with a
> prompt I can paste back to the pup-bridge thread documenting the exact verb contract (name,
> args, response shape, min ab version) so pup can wire it.
Deferred with it: the `UserPromptSubmit`/`PreToolUse` hook mailbox, the inotify instant-push
watcher, the thread-naming convention (`claude -n` == `--ai-thread`), and the session-monitor
rail for idle threads.

## Phase 3 — move-to-webview ("Send back to my workspace") — DEFERRED (future TODO, John 2026-08-10)

- **ah (local)**: pup → ab SDK callback direct API → `hd_api {port:47083, POST
  /api/workspaces/editor/local/workspace/current/tabs}` with the webview panelType +
  `initialState.url`, open-or-refresh semantics. Then close the pup window (its owner initiated
  the move). Pill + jump-list action: "Move to my workspace". Endpoint defaulted from the
  window's recorded caller container; picker only when unresolved.
- **hc (cloud)**: desktop → `https://hydrogen.adom.inc/api/workspaces/editor/{owner}/{repo}/
  current/tabs` (public HTTPS; slug→owner/repo via carbon lookup, cached). BLOCKED on one ask:
  accept ab's Adom Cloud session token for workspace mutations (today only the container-mounted
  api-key is documented). Surface the E2 409 (editor tab not live) honestly; rewrite
  localhost URLs to the slug proxy URL (SSRF guard).
- **ab asks to file** (nice-to-have, not blockers): `hydrogen_endpoints` (union of ah + connected
  hc with owner/repo/liveness) and `hydrogen_open_webview {endpoint,url,name}`; mirror Browser
  Picker destination ids so hbw's move-to-surface and pup's are one concept.

## Step 0 (on approval): persist this plan in the pup repo

Copy this document to `docs/PLAN-interactive-surface.md` in
`/home/adom/project/adom-desktop-puppeteer-bridge/`, push to the wiki repo (source-only, like
dev-skills), and keep it updated as phases land (status line per phase). The repo, not the local
plans dir, is the permanent home (John).

## Files touched
- `src/server.js` (cursors, tight-crop, quick capture, clipboard helper, USER_CALLERS refactor,
  settings, move-to-webview verb), new `src/extension/` (MV3: manifest, popup, service worker,
  icons — shipped in the zip; pup pushes it to the desktop for Load-unpacked),
  `SKILL.md` (teach toolbar, quick capture, clipboard sidecar, move-to-webview),
  `docs/PLAN-interactive-surface.md` (this plan, step 0),
  issues on adom/adom-bridge + hydrogen/cloud for the phase-3 asks and the deferred
  container_deliver TODO.

## Verification
- Phase 1: ralph the toolbar path — fire each popup action against pup's /command with caller
  'user-toolbar' (the exact fetches the popup makes) and assert the same observable effects as
  the jump-list ralph; screenshot a window during annotate to verify cursors switch per tool and
  the toast preview renders (read the pixels, per the audit doctrine).
- Phase 2: simulate a quick-capture mouse-up on a test window; assert the PNG exists with iTXt
  sidecar (sharp metadata read), the clipboard holds BOTH formats (PowerShell:
  Get-Clipboard -Format Image non-null AND Get-Clipboard text contains the JSON header), and the
  toast thumbnail matches the crop. Then John's live test: draw, paste into a real chat.
- Phase 3 (ah): move a pup window to a local Hydrogen webview on the winvm or John's ah, assert
  tab exists via `GET /workspace/tabs`, pup window closed, action logged with delegate
  attribution. hc leg only after the auth ask lands.