adom-desktop API patterns (chip-fetcher playbook)

Trusted clicks, popup auto-tracking, smart-pick, browser_fetch_url, pull_file. Read this when you need to drive a pup browser tab beyond the basic browser_navigate + browser_eval couplet.

Trusted click via browser_input_dispatch (adom-desktop ≥ v1.4.7)

When a click "lands but does nothing" — that's almost always because the framework / vendor JS gates the handler on event.isTrusted, and page-side dispatchEvent(new MouseEvent(...)) is isTrusted: false. browser_input_dispatch routes through Chromium CDP Input.dispatchMouseEvent / dispatchKeyEvent — events have isTrusted: true, so framework handlers (React/Vue/Svelte) and most vendor anti-automation gates actually fire.

# Click by selector (preferred — uses element bounding rect + scrollIntoView)
adom-desktop browser_input_dispatch '{
  "sessionId":"chip-fetcher",
  "type":"click",
  "selector":"button.wg-button--primary"
}'

# Click by viewport coords
adom-desktop browser_input_dispatch '{
  "sessionId":"chip-fetcher",
  "type":"click",
  "x":860, "y":803
}'

# Type into an input
adom-desktop browser_input_dispatch '{
  "sessionId":"chip-fetcher",
  "type":"type",
  "text":"DMG2305UX-7",
  "selector":"input[name=search]"
}'

# Press a key
adom-desktop browser_input_dispatch '{
  "sessionId":"chip-fetcher",
  "type":"key",
  "key":"Enter"
}'

# Mouse move (human-like trajectory before a click)
adom-desktop browser_input_dispatch '{
  "sessionId":"chip-fetcher",
  "type":"move",
  "x":400, "y":500, "steps":10
}'

Optional fields: tabId (defaults to active tab), button (left|right|middle), clickCount (2 for double-click), delay (ms between mousedown/mouseup or between keystrokes).

Use browser_input_dispatch when:

  • Vendor "Generate" / "Download" / "Submit" buttons no-op on browser_eval clicks
  • Login forms behind framework gates
  • Modal dismissal needs key:"Escape" and DOM-level dispatch fails
  • Any time you'd say "I clicked it but nothing happened"

browser_eval-side click is still fine for buttons that DO fire (most CSE Download Model buttons, simple modal openers). browser_input_dispatch is the upgrade path when those silently no-op.

Tabs the page itself spawns (window.open(), target="_blank" form submits, <a target="_blank"> clicks) used to be invisible. Now they auto-track:

adom-desktop browser_list_tabs '{"sessionId":"chip-fetcher"}'
# → tabs: [
#     { tabId:"tab-1", opener:"user",  openerTabId:null,    url:"..." },
#     { tabId:"tab-2", opener:"popup", openerTabId:"tab-1", url:"..." }
#   ]

opener:"user" = chip-fetcher opened it. opener:"popup" = the page spawned it; openerTabId names the tab that triggered the popup.

Popups don't auto-focus — use browser_switch_tab to make one active before screenshots, or pass tabId to browser_eval / browser_screenshot to target it without switching.

Clicks may open the result in a NEW TAB

Every "Download Datasheet" / "Generate" / "Open PDF" / "View Document" click on a vendor site can spawn a new browser tab. Common sites where this fires: WAGO ("Generate Datasheet" → PDF in new tab), Microchip, ST, Renesas, ADI, even some TI app-note rows.

The rule, applied after every click that "should" produce a download/PDF/CAD file:

  1. Wait ~2-5s for the popup to fully load (it may be a redirect chain).
  2. browser_list_tabs — look for a new entry with opener:"popup". The openerTabId confirms which tab triggered it.
  3. browser_eval (or browser_screenshot) with tabId set to the popup's tabId — interact with the popup directly without switching the active tab.
  4. For a PDF popup: grab location.href from the popup, then use browser_fetch_url (Mode A or B). Do NOT use in-page fetch(location.href) on a Chrome PDF Viewer popup — it returns the PDF Viewer HTML wrapper (~200 KB stub), not the binary.
  5. browser_close_tab with the popup's tabId to clean up after.

This also applies to "Buy" → distributor checkout, "View 3D" → embedded viewer in new tab, "Open Application Note" → PDF in new tab.

Direct user pushback: "wago opens the datasheet in a new tab, so you could just see that tab and then save it locally" + "add that to the skill that this could happen on any website."

Smart-pick (adom-desktop ≥ v1.4.9)

When a selector matches multiple elements, the relay no longer silently picks document-order first. It filters to visible+text matches, detects open modals, and prefers candidates inside the topmost modal (with area tiebreak).

Always check the response fields:

{
  "ok": true,
  "matchedCount": 4,
  "chosenIndex": 3,
  "clickedRect": {"x":890,"y":803,"w":193,"h":48},
  "clickedText": "Generate data sheet",
  "pickStrategy": "modal-scoped-largest",
  "modalDetected": true,
  "insideModal": true
}

pickStrategy values to act on:

  • only-match / first-match — single match, or you opted out via firstMatch:true
  • modal-scoped-largest — modal detected, picked inside it (preferred)
  • visible-text-largest — no modal, area-tiebreak among visible+text
  • visible-text-largest-no-modal-match — modal detected but selector hit nothing inside it. Either your selector is wrong, OR modal-detection misfired (Vue-Toastification false-positive bug).
  • visible-largest — no text matches, fell back to area
  • fallback-first-document-order — nothing visible at all

If pickStrategy is visible-text-largest-no-modal-match and you can SEE the button visually, refine the selector OR pass {x, y} coords directly. Always sanity-check clickedText against expectation before assuming success.

Vue-Toastification false-positive workaround

Pages using Vue-Toastification (WAGO and others) ship empty <div class="Vue-Toastification__container"> elements at position:fixed; z-index:9999; height:100vh. Smart-pick's >25%-viewport heuristic sees them as "modal." Result on WAGO: real Generate-Datasheet modal is missed, smart-pick reports modalDetected:true insideModal:false pickStrategy:"visible-text-largest-no-modal-match", and falls back to area-tiebreak among page-level matches → clicks "Add to shopping cart" instead of "Generate data sheet."

Workaround: when pickStrategy === "visible-text-largest-no-modal-match" AND modalDetected === true AND the user can clearly see the button on screen, fall back to coords:

RECT=$(adom-desktop browser_eval "{\"sessionId\":\"$S\",\"expr\":\"(()=>{const b=document.querySelector('YOUR-SELECTOR'); if(!b)return null; const r=b.getBoundingClientRect(); return JSON.stringify({x:Math.round(r.left+r.width/2),y:Math.round(r.top+r.height/2)});})()\"}" | jq -r '.result')
adom-desktop browser_input_dispatch "{\"sessionId\":\"$S\",\"type\":\"click\",\"x\":$(echo $RECT | jq .x),\"y\":$(echo $RECT | jq .y)}"

Modal-detection isPlausibleModal filter is fixed in v1.4.11 — empty containers correctly skipped.

browser_fetch_url — pull bytes through the session (adom-desktop ≥ v1.4.9)

When a popup tab serves application/pdf, in-page fetch(location.href) returns Chrome's PDF Viewer HTML wrapper, NOT the binary. browser_fetch_url re-issues the request through the bridge's session context, bypassing the page render layer. This is the canonical way to grab popup-PDF bytes.

Three modes:

# Mode A — container-side write (preferred for chip-fetcher's library/)
adom-desktop browser_fetch_url '{
  "sessionId":"chip-fetcher",
  "tabId":"tab-2",
  "url":"https://wago.priintcloud.com/datasheets/2601-3105/en",
  "saveTo":"/home/adom/project/chip-fetcher/library/WAGO-2601-3105/datasheet.pdf"
}'
# → {ok:true, savedTo:"/home/.../datasheet.pdf", savedToFilesystem:"container",
#    bytes:3798336, contentType:"application/pdf", status:200}

# Mode B — desktop-side write (drop a CAD bundle in user's Downloads)
adom-desktop browser_fetch_url '{
  "sessionId":"chip-fetcher",
  "tabId":"tab-2",
  "url":"https://...",
  "desktopSaveTo":"C:\\Users\\john\\Downloads\\bundle.zip"
}'

# Mode C — neither saveTo nor desktopSaveTo → bodyBase64 in response
#   (caller decides where to write; useful for files <50 MB)

Combined recipe — vendor "Generate Datasheet" → popup → grab PDF

SESSION="chip-fetcher"
URL="https://www.wago.com/global/p/2601-3105"
OUT="/home/adom/project/chip-fetcher/library/WAGO-2601-3105/datasheet.pdf"

# 1. Open product page
adom-desktop browser_navigate "{\"sessionId\":\"$SESSION\",\"url\":\"$URL\"}"
sleep 6

# 2. Auto-accept cookies
adom-desktop browser_eval "{\"sessionId\":\"$SESSION\",\"expr\":\"<auto-accept-cookies-snippet>\"}"
sleep 2

# 3. Open the modal (eval-click is fine for non-trusted modal openers)
adom-desktop browser_eval "{\"sessionId\":\"$SESSION\",\"expr\":\"Array.from(document.querySelectorAll('button, a')).find(e => /^generate datasheet$/i.test((e.textContent||'').trim())).click()\"}"
sleep 3

# 4. TRUSTED click on modal submit
adom-desktop browser_input_dispatch "{\"sessionId\":\"$SESSION\",\"type\":\"click\",\"selector\":\"button.wg-button--primary\"}"
sleep 5

# 5. Capture popup URL FROM list_tabs IMMEDIATELY (popup tabs auto-close)
LIST=$(adom-desktop browser_list_tabs "{\"sessionId\":\"$SESSION\"}")
POPUP_URL=$(echo "$LIST" | jq -r '.tabs[] | select(.opener=="popup") | .url' | head -1)
POPUP_TAB=$(echo "$LIST" | jq -r '.tabs[] | select(.opener=="popup") | .tabId' | head -1)

# 6. Fetch PDF bytes through the session, save container-side
adom-desktop browser_fetch_url "{
  \"sessionId\":\"$SESSION\",
  \"tabId\":\"$POPUP_TAB\",
  \"url\":\"$POPUP_URL\",
  \"saveTo\":\"$OUT\"
}"

tab_not_found recovery (v1.4.11+)

When the Chrome PDF Viewer popup auto-closes between your browser_list_tabs and a follow-up browser_eval tabId:popupTabId, the structured error returns lastKnownUrl directly:

{
  "ok": false,
  "errorCode": "tab_not_found",
  "lastKnownUrl": "https://example.com/",
  "openerTabId": "tab-1",
  "_hint": "...Call browser_fetch_url with that URL (and the opener tabId for cookies) to get its bytes — do NOT try to eval against the closed tabId."
}
# Canonical popup-PDF recovery pattern
RESP=$(adom-desktop browser_eval "{\"sessionId\":\"X\",\"tabId\":\"$POPUP\",\"expr\":\"1\"}" 2>&1 || true)
if echo "$RESP" | jq -e '.errorCode == "tab_not_found"' >/dev/null; then
  URL=$(echo "$RESP" | jq -r '.lastKnownUrl')
  OPENER=$(echo "$RESP" | jq -r '.openerTabId')
  adom-desktop browser_fetch_url "{\"sessionId\":\"X\",\"tabId\":\"$OPENER\",\"url\":\"$URL\",\"saveTo\":\"…\"}"
fi

adom-desktop pull_file (streaming, v1.4.3+)

pull_file streams files in 1 MiB binary WS frames straight to disk — no base64 inflation, no JSON-per-chunk overhead, incremental SHA256 verification, 600 s per-file timeout. Works for files of any size we've seen (75 MB NXP Reference Manuals complete in ~12 s).

adom-desktop pull_file '{
  "filePaths": ["C:/Users/<USER>/Downloads/<file>"],
  "saveTo": "/home/adom/project/chip-fetcher/incoming"
}'
# → {"success": true, "files": [{"name": "...", "path": "/home/adom/...", "size": ..., "sha256": "...", "chunks": ...}]}

If chunks is missing from the response, you're hitting an old non-streaming path — see "Stale relay" below.

Stale-relay gotchas

adom-desktop serve (container-side)

The adom-desktop serve process holds its own argv[0] binary in memory; on-disk upgrades don't reload it. Symptom: Unknown action: send_and_wait_streamed from the relay's HTTP API even though the on-disk binary is current. Fix:

pkill -f "adom-desktop serve"
sleep 1
nohup /usr/local/bin/adom-desktop serve > /tmp/adom-desktop-serve.log 2>&1 &
disown
adom-desktop --version

Stale Puppeteer bridge (node.exe on Windows desktop)

adom-desktop's Windows-side puppeteer bridge (node.exe) is auto-spawned at first browser command and stays resident. Upgrading the desktop app reinstalls plugins/puppeteer/server.js on disk, but the existing node.exe keeps its old code loaded. Symptom: relay rejects new commands with "Unknown command: <name>" even though the binary's --version reports the new version.

ssh user@host "taskkill //f //im node.exe"
# then send any browser command to auto-respawn
adom-desktop browser_status '{"sessionId":"chip-fetcher"}'

Verification command after any adom-desktop upgrade:

adom-desktop browser_input_dispatch '{"sessionId":"x","type":"click","x":1,"y":1}'
# Expected: {"ok":true,"dispatched":"click",…} or session-not-found
# WRONG: "Unknown command: browser_input_dispatch" — bridge is stale, kill node.exe.

The first thing chip-fetcher does on any new page is dismiss the cookie banner with "Accept All" — without being told, without asking the user, without thinking about it. Cookie banners block clicks (z-index gate) and break programmatic navigation.

// snippet to use via browser_eval after every navigation
(()=>{
  const tries = (root) => {
    const btns = Array.from(root.querySelectorAll("button, a"));
    const re = /^(accept all|accept all cookies|accept|i agree|allow all|ok)$/i;
    const m = btns.find(b => re.test((b.textContent||"").trim()));
    if (m) { m.click(); return true; }
    return false;
  };
  if (tries(document)) return "accepted";
  // Walk shadow roots — Usercentrics et al. use them
  const hostsAttempted = new Set();
  const walk = (root) => {
    for (const el of root.querySelectorAll("*")) {
      if (el.shadowRoot && !hostsAttempted.has(el)) {
        hostsAttempted.add(el);
        if (tries(el.shadowRoot)) return true;
        if (walk(el.shadowRoot)) return true;
      }
    }
    return false;
  };
  return walk(document) ? "accepted-in-shadow" : "no-banner";
})()

Direct user quote: "you're supposed to always accept their stupid cookie dialogs automatically". Never click "Reject", "Decline", "Manage preferences", or "Save settings" — those are slower and the user has explicitly ruled out anything but Accept All.

Comprehensive page killer (cookies + surveys + modals)

Run this AFTER every browser_navigate, AND a second time after a 2-second delay (some popups load after first paint):

const acceptRegex = /^(accept all|accept|allow all|i accept|got it|agree|ok|continue|yes)$/i;
const declineRegex = /^(not today|no thanks|maybe later|decline|close|dismiss|not now|no, thanks|cancel|skip|later)$/i;
Array.from(document.querySelectorAll("button, a, [role=button]"))
  .filter(b => {
    const t = b.textContent.trim();
    return acceptRegex.test(t) || declineRegex.test(t);
  })
  .forEach(b => b.click());
// Click close X icons (cookie banners, modals, popups)
document.querySelectorAll("[aria-label*=close i], [class*=banner-close], [class*=modal-close], [class*=popup-close], #onetrust-close-btn-container button, #onetrust-accept-btn-handler, #surveyModal .close, [id*=survey] [class*=close]").forEach(b => b.click());
// Hide any survivors
document.querySelectorAll("[id*=survey], [class*=survey-modal], [class*=popup-modal], [aria-modal=true]:not([id^=mvOverlay])").forEach(el => el.style.display = "none");
document.body.style.overflow = "";

NXP, ST, ADI, and Microchip all have multiple modal overlays that compound: cookie banner + privacy banner + survey + chat widget.

Pup profile name is SACRED

Use profile:"chip-fetcher" and only profile:"chip-fetcher". Never chip-fetcher-2, chip-fetcher-temp, cf-bust, or any other variant. Each pup profile is an isolated Chrome user-data directory at <adom-desktop>/plugins/puppeteer/profiles/<name>/ — switching names dumps you into a fresh profile with no cookies, no signed-in vendor accounts, no work-Chrome sync, no CSE login, nothing.

Do not rotate profiles to bust Chrome's HTML cache. That destroys vendor logins. Bust the cache the right way:

  • Add a ?v=$(date +%s) query string when reloading via location.reload() or browser_navigate.
  • Run chip-fetcher serve on a different port and reopen — the proxy treats it as a new origin.
  • Send Cache-Control: no-store headers from the chip-fetcher viewer's HTTP server.

If you've already broken the profile (vendor logins gone, Google account avatar missing in upper-right) the only recovery is the user re-signs into the work Chrome profile. Treat profile-rotation regressions as a P0 skill bug.

Don't pile up pup tabs

Reuse the existing tab. Don't open a new tab for every navigation. Use browser_navigate (or location.href = … via browser_eval) when moving between part pages, search pages, or vendor sites. Only call browser_open_tab when you genuinely need two pages side-by-side. After a download fires and the zip is auto-pulled, close stale tabs with browser_close_tab.

Direct user quote: "never open too many tabs cuz i'm seeing way too many". Tab clutter is also a perf hit — every chip-fetcher tab carries cookies, cached JS, and an open Cloudflare check, and after ~10 the pup window starts thrashing.

adom-desktop commands you'll actually use

# Drive a browser tab
adom-desktop browser_navigate '{"sessionId":"chip-fetcher","url":"..."}'
adom-desktop browser_eval '{"sessionId":"chip-fetcher","expr":"..."}'
adom-desktop browser_screenshot '{"sessionId":"chip-fetcher","maxWidth":900}'
adom-desktop browser_raise_os_window '{"sessionId":"chip-fetcher"}'

# Run shell on user's desktop (requires approval — first time per session)
adom-desktop shell_execute '{"command":"<cmd>","timeoutSeconds":30}'

# Pull a file from the user's desktop into the container
adom-desktop pull_file '{
  "filePaths": ["C:\\Users\\<USER>\\Downloads\\<file>"],
  "saveTo": "/home/adom/project/chip-fetcher/incoming"
}'