Pup - the AI's own browser

@@ -0,0 +1,198 @@+# pup Interactive Surface: toolbar extension, drawing cursors, snip return path, move-to-webview++**Status (update as phases land):** approved 2026-08-10 · phase 1 IN PROGRESS · phase 2 in+progress (clipboard rail; container push deferred) · phase 3 pending · extension render-first+pending John's sign-off++## Context++Annotate + snip shipped in pup 2.0.49 (overlay + jump-list task; capture stays in-DOM so the+owning thread's next screenshot shows it). John's follow-ups define a coherent feature set:+actions must be discoverable IN the browser (not just jump lists), drawing needs real tool+cursors, a Done'd snip must reach the owning AI thread's chat, and a pup window should be able to+move back to a webview in the workspace it came from. Exploration (3 agents) resolved the+architecture questions; this plan is the build order.++**Headline decisions**+- **Toolbar lives in Chrome's TOP BAR via a small extension** (John: "menu item in the chrome bar+  up at the top rather than an overlay in their page content... like Claude Desktop's"). Browser+  chrome is unreachable from CDP, so an MV3 extension action icon is the only way to put a button+  there — and it's the RIGHT shape: the extension is a thin trigger surface; pup stays the engine+  (extension popups may fetch pup's loopback `/command` with host_permissions — the mixed-content+  wall only blocks web pages). The in-page pill is DROPPED.+- **Return path = CLIPBOARD-FIRST** (John's pick): on mouse-up the tight-cropped PNG lands in+  the Windows clipboard as image + text/plain JSON sidecar together (multi-format DataObject), so+  one paste into any AI chat carries both; the PNG also bakes a caption bar + iTXt metadata so an+  image-only paste still carries context. Push-back-to-container via ab: DEFERRED (TODO, ab+  prompt ready in phase 2).+- **Move-to-webview: build the local Hydrogen (ah) leg NOW** (zero new ab verbs needed); hc leg+  is blocked on one cloud-side auth decision (filed as an ask).++## Phase 1 — pup 2.0.50: drawing cursors, quick capture + the toolbar extension++All in `/home/adom/project/adom-desktop-puppeteer-bridge/src/server.js`.++**1a. Photoshop-style tool cursors** (in `ANNOTATE_OVERLAY_JS`, ~L5260):+- Inline-SVG data-URI cursors, hotspot at the working tip: pen → pencil glyph with a live swatch+  of the current color (cursor rebuilt on color change); arrow → fletched pointer; box + snip →+  precise crosshair; text → I-beam. No external assets (CSP-safe).++**1a2. Tight-crop annotation captures (John's ask — beat Claude Desktop on tokens)**:+- On Done, compute the bounding box of ALL drawn strokes (the overlay already holds every+  stroke's points), pad by a ~48px margin, clamp to the viewport, and capture ONLY that clip —+  not the whole browser view ("if the user wants the whole view, they'll draw a circle around+  it"). Text notes count toward the box (measure the rendered text width). Snip is unchanged+  (already exact-region). The full-view drawing still stays in-DOM for the owning thread's+  next full screenshot, so no information is lost — the CAPTURED PNG is just the tight,+  token-cheap version that rides the return path.++**1a3. Quick capture on mouse-up (John's ask)** — an "⚡ Auto-send" toggle in the overlay toolbar+(also the default mode when launched via ▣ Snip or the hotkey):+- **The moment the mouse button lifts**: capture fires immediately (tight-crop rules from 1a2) —+  no Done click. If the user keeps drawing, each subsequent mouse-up recaptures with the union+  bounding box and supersedes the previous capture (single circle = instant; multi-stroke = last+  lift wins with everything included; dedupe by capture id downstream so the thread sees one).+- **Clipboard**: the PNG is copied to the Windows clipboard at the same instant (via ab clipboard+  verb if present, else PowerShell `Set-Clipboard` on the saved file) — fastest path is often the+  user pasting straight into whatever chat window they're in.+- **Confirmation toast WITH preview**: instantly on mouse-up, an in-page toast (bottom-right,+  same visual family as the toolbar) shows "▣ Snippet copied to your clipboard · sent to+  <thread>" WITH a thumbnail of the ACTUAL captured PNG (data-URL preview) — the user sees their+  red circle and the margin that was grabbed, verifying the crop got what they wanted before they+  tab away. Auto-dismisses ~4s; click-to-dismiss; a "↻ recapture" link if the crop missed.+- Phase 2's clipboard write (image + sidecar) fires from this same mouse-up event.++**1b. The "Adom Pup" toolbar extension** — a button in Chrome's TOP BAR (the extension action+area right of the omnibox), Claude-Desktop-style:++```+  ← →  ⟳ │ wiki.adom.inc/adom/adom-bridge          │ ⧉ │ [⠿]  ←  Adom action icon+                                                          ╭──────────────────────────╮+                                                          │ ✎ Annotate               │+                                                          │ ▣ Snip a region          │+                                                          │ ◐ Switch to public view  │+                                                          │ ▦ Open pup dashboard     │+                                                          │ ⓘ Owned by: kicad-dev    │+                                                          │ ⇱ Move to my workspace   │  (phase 3)+                                                          │ ✕ Close this window      │+                                                          ╰──────────────────────────╯+```++- **MV3 extension, thin trigger surface only**: the popup fetches pup's loopback+  `http://127.0.0.1:<port>/command` directly (host_permissions `http://127.0.0.1/*` — extensions+  are exempt from the mixed-content wall that blocks web pages). Port discovery: probe the+  known pup port range against `/health` (`bridge:"puppeteer"` fingerprint), cache in+  chrome.storage. All engine logic stays in pup; the popup sends the SAME verbs the jump lists+  send, with caller `'user-toolbar'` (added to the new `USER_CALLERS` exemption set alongside+  `'user-taskbar-menu'`, refactoring the 4 exact-string compares at L10463/L10491/L10510/L11579).+- **Per-tab owner badge**: the action icon carries a small colored badge per tab+  (`chrome.action.setBadgeText({tabId})`) with the owning thread's initials/color, pulled from+  pup `/status` — the "whose window is this" signal right in the toolbar.+- **Keyboard command**: `Ctrl+Shift+A` → snip pre-armed (chrome.commands), the Win+Shift+S muscle+  memory John described.+- **Loading strategy (the one hard constraint, verified 2026-08-10)**: branded Chrome ≥137+  REMOVED `--load-extension`; it still works on Chrome for Testing/Chromium. The CDP replacement+  (`Extensions.loadUnpacked` / Puppeteer `browser.installExtension`) requires launching with+  `--remote-debugging-pipe`, which conflicts with pup's port-based session recovery (recovery+  reconnects via DevToolsActivePort after a bridge respawn). So:+  1. **Now (John/dev)**: pup pushes the unpacked extension dir to the desktop; ONE-TIME manual+     "Load unpacked" into the pup profile (chrome://extensions, dev mode). Persists in the+     durable adom-you profile across every launch. Dev-mode startup bubble is the known cost.+  2. **Fleet**: publish "Adom Pup" UNLISTED on the Chrome Web Store (the org already has store+     presence via nb, the native-browser extension); pup's welcome flow deep-links it — one click+     per profile, no dev-mode nag.+  3. **Later investigation**: pipe-based launch + `installExtension` for zero-click automation,+     ONLY if recovery can be made pipe-compatible (do not risk the recovery architecture for it).+- **Settings (explicit user toggle, John's ask)**: a visible row in pup's dashboard Settings+  dialog — `toolbarExtension` master toggle (default on) with sub-toggles `tbOwnerBadge` (the+  per-tab thread badge) and `tbSnipHotkey` (Ctrl+Shift+A), wired exactly like the jump-list+  family (SETTINGS_DEF row + set-setting whitelist + `_chosen` stamping + live apply: OFF stops+  the badge feed and tells the extension to disable itself via its next /status poll; ON+  re-enables without a browser restart). The annotate feature keeps its own existing `jlAnnotate`+  toggle. Jump lists stay: same actions in both places. The in-page pill is dropped.+- **First deliverable is render-only**: the popup UI built + loaded in a test profile,+  screenshot for John's sign-off before wiring actions.++## Phase 2 — pup 2.0.51: the clipboard-first return path (container push DEFERRED)++**Primary rail (John's pick): the clipboard, with a JSON sidecar that survives pasting.**+On quick-capture mouse-up (or Done):+1. Tight-crop PNG (1a2) is finalized with:+   - a **baked caption bar** (thin strip under the crop: `pup · <thread>'s window · <url> · <ts>`)+     — visible to ANY AI the image is pasted into, in any chat product; toggleable in settings;+   - an **iTXt metadata chunk** carrying the full JSON sidecar {thread, container, sessionId,+     url, ts, shapes, viewport} for tooling that reads PNG metadata (sharp, already a dep,+     handles both).+2. **Multi-format clipboard write**: image + text/plain set TOGETHER via one .NET `DataObject`+   (small PowerShell -STA helper pup spawns on the desktop; plain Set-Clipboard cannot do+   multi-format). The text/plain payload is a one-line human-readable header + the JSON sidecar.+   Pasting into Claude/Codex/Kimi chat inputs attaches the image AND inserts the text (browsers+   expose both items to the paste event); custom mime formats do NOT survive into web inputs, so+   plain text is the only sidecar format used.+3. Toast (1a3) already confirms with the preview thumbnail; add "sidecar: copied" line.+4. Local archive under `SHOTS_DIR` + `session._lastAnnotation` stays (owning thread can still+   fetch via pull_file when asked).++**DEFERRED (TODO — revisit after the clipboard rail proves out): push-back-to-container via ab.**+Too complex for now (John). When revived: ONE new ab verb over the existing relay ws —+`container_deliver {server, files, note, inbox}` — then the hook mailbox + inotify instant-push+design below. The ready-to-paste AI prompt for the ab build thread (John's workflow: paste to ab,+ab replies with a contract prompt to paste back to pup):+> pup needs a reverse-delivery verb: the user annotates/snips a page in a pup window on the+> desktop, and the captured PNG + a JSON note must reach the cloud container whose AI thread owns+> that window. You already hold the connection: each container's relay server is a ws ab dials.+> Add `container_deliver {server, files:[{path}|{b64,name}], note, inbox}`: desktop-side ab+> streams the files over that ws to the container-side relay, which writes them to+> `~/.adom/inbox/<inbox>/<ts>-<id>/` (files + note.json) and acks. Size cap ~15 MB, errors honest+> (server not connected, disk). Called by pup via the SDK callback channel with caller-identity+> headers, so log it in the activity log like any verb. When it's built and shipped, reply with a+> prompt I can paste back to the pup-bridge thread documenting the exact verb contract (name,+> args, response shape, min ab version) so pup can wire it.+Deferred with it: the `UserPromptSubmit`/`PreToolUse` hook mailbox, the inotify instant-push+watcher, the thread-naming convention (`claude -n` == `--ai-thread`), and the session-monitor+rail for idle threads.++## Phase 3 — move-to-webview ("Send back to my workspace") — ah leg IN SCOPE NOW++- **ah (local)**: pup → ab SDK callback direct API → `hd_api {port:47083, POST+  /api/workspaces/editor/local/workspace/current/tabs}` with the webview panelType ++  `initialState.url`, open-or-refresh semantics. Then close the pup window (its owner initiated+  the move). Pill + jump-list action: "Move to my workspace". Endpoint defaulted from the+  window's recorded caller container; picker only when unresolved.+- **hc (cloud)**: desktop → `https://hydrogen.adom.inc/api/workspaces/editor/{owner}/{repo}/+  current/tabs` (public HTTPS; slug→owner/repo via carbon lookup, cached). BLOCKED on one ask:+  accept ab's Adom Cloud session token for workspace mutations (today only the container-mounted+  api-key is documented). Surface the E2 409 (editor tab not live) honestly; rewrite+  localhost URLs to the slug proxy URL (SSRF guard).+- **ab asks to file** (nice-to-have, not blockers): `hydrogen_endpoints` (union of ah + connected+  hc with owner/repo/liveness) and `hydrogen_open_webview {endpoint,url,name}`; mirror Browser+  Picker destination ids so hbw's move-to-surface and pup's are one concept.++## Step 0 (on approval): persist this plan in the pup repo++Copy this document to `docs/PLAN-interactive-surface.md` in+`/home/adom/project/adom-desktop-puppeteer-bridge/`, push to the wiki repo (source-only, like+dev-skills), and keep it updated as phases land (status line per phase). The repo, not the local+plans dir, is the permanent home (John).++## Files touched+- `src/server.js` (cursors, tight-crop, quick capture, clipboard helper, USER_CALLERS refactor,+  settings, move-to-webview verb), new `src/extension/` (MV3: manifest, popup, service worker,+  icons — shipped in the zip; pup pushes it to the desktop for Load-unpacked),+  `SKILL.md` (teach toolbar, quick capture, clipboard sidecar, move-to-webview),+  `docs/PLAN-interactive-surface.md` (this plan, step 0),+  issues on adom/adom-bridge + hydrogen/cloud for the phase-3 asks and the deferred+  container_deliver TODO.++## Verification+- Phase 1: ralph the toolbar path — fire each popup action against pup's /command with caller+  'user-toolbar' (the exact fetches the popup makes) and assert the same observable effects as+  the jump-list ralph; screenshot a window during annotate to verify cursors switch per tool and+  the toast preview renders (read the pixels, per the audit doctrine).+- Phase 2: simulate a quick-capture mouse-up on a test window; assert the PNG exists with iTXt+  sidecar (sharp metadata read), the clipboard holds BOTH formats (PowerShell:+  Get-Clipboard -Format Image non-null AND Get-Clipboard text contains the JSON header), and the+  toast thumbnail matches the crop. Then John's live test: draw, paste into a real chat.+- Phase 3 (ah): move a pup window to a local Hydrogen webview on the winvm or John's ah, assert+  tab exists via `GET /workspace/tabs`, pup window closed, action logged with delegate+  attribution. hc leg only after the auth ask lands.