Pup - the AI's own browser

@@ -0,0 +1,156 @@+---+name: pup-bot-walls+description: "DEV skill (source-only). What to do when a site serves pup a bot wall / CAPTCHA / 'Verification Required' instead of the page — identifying WHICH wall it is (DataDome vs Cloudflare vs Akamai vs Imperva), what pup already does about each, the sourcing ladder that avoids the fight entirely (official API → nb → pup), and how to hand a challenge to the human who is driving. Read this before touching the consumer-UA override, before adding retries to a crawl, and before anyone proposes fingerprint spoofing. Trigger words: bot wall, captcha, verification required, datadome, cloudflare challenge, just a moment, akamai, imperva, incapsula, blocked by mouser, blocked by digikey, chip-fetcher blocked, crawl blocked, are you spoofing the user agent, pup user agent, scraping blocked, access denied, 403 on a vendor site."+---++# Bot walls: identify it, avoid the fight, hand the hard ones to the human++## The doctrine John set (2026-08-20)++> *"we want pup to be able to be used for chip-fetcher which will crawl lots of websites via pup...+> truly you are being guided by a human, me right now, but any adom user in the future, and you are+> just an assistant for the human, so you are a human from datadome's and mouser's perspective.+> its just a new world with ai-first apps like adom is now."*++He is right about the substance, and the substance sets the ENGINEERING rule, not a licence:++- A pup session runs **on the user's own machine, in their own browser profile, at their explicit+  direction**. There is a real human on the other end of every request. That is genuinely different+  from a datacenter scraper farm, and it is why "we look like a bot" is usually a FALSE POSITIVE.+- So the goal is **stop earning the false positive**, not **defeat the detector**. Those look similar+  from one angle and are completely different engineering. Behaving like a human-directed session —+  sane pacing, one page at a time, warm cookies, real input, using the front door where one exists —+  lowers your score honestly and holds up. Fingerprint spoofing is an arms race you re-lose on every+  vendor update, and it makes every future false positive harder to diagnose because you can no+  longer trust what you are presenting.+- **We do not solve CAPTCHAs on the human's behalf.** A CAPTCHA is a site asking "is a person+  there?" and in Adom the honest answer is *yes, and here they are* — surface it, let them click it.+  That is not a limitation to engineer around; it is the correct answer to the question being asked.++## Step 1 — identify WHICH wall (never guess; they need different answers)++Read the challenge page's own markup. `pup_eval` on the stuck session:++```js+JSON.stringify({+  title: document.title,+  len: (document.body.innerText || '').length,+  scripts: [...document.scripts].map(s => s.src).filter(Boolean).slice(0, 6),+  head: document.documentElement.outerHTML.slice(0, 500)+})+```++| Wall | Tell | Notes |+|---|---|---|+| **DataDome** | `ct.captcha-delivery.com/c.js`, inline `var dd={'rt':'c','cid':…,'t':'fe'}` | Behavioural + fingerprint + IP velocity. Does NOT care what your UA claims. **Mouser runs this.** |+| **Cloudflare** | "Just a moment…", `/cdn-cgi/challenge-platform/` | Gates heavily on the **Sec-CH-UA brand list** — this is the one pup's consumer-UA override already solves. |+| **Akamai** | `_abck` / `bm_sz` cookies, `/akam/` script paths | Also brand-list sensitive; same override helps. |+| **Imperva/Incapsula** | `_Incapsula_Resource`, "Request unsuccessful" | Usually IP/ASN reputation. |++A blank body with a tiny `htmlLen` (~1.6 KB) and a title that is just the bare domain is the classic+"challenge page, not your content" signature. `len: 0` with `title: "mouser.com"` is what a wall+looks like — do not mistake it for a slow load.++## Step 2 — what pup ALREADY does, and what it does NOT cover++`computeConsumerUA()` / `applyConsumerUA()` in `server.js` override the UA header **and**+`navigator.userAgentData` together, so every page sees consumer **Google Chrome** rather than+Chrome for Testing. Measured live on a real window (2026-08-20):++```+userAgent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) … Chrome/148.0.0.0 Safari/537.36+webdriver : false                       (--enable-automation stripped via ignoreDefaultArgs)+brands    : "Not/A)Brand" 99, "Google Chrome" 148, "Chromium" 148+```++**This is not spoofing a different device.** It is the same Windows Chrome the user runs, correctly+declaring itself as Google Chrome instead of as a test build. It fixed Cloudflare/Akamai-class walls+(DigiKey, Arrow, SnapEDA, CSE/UL, Octopart all went from "Just a moment…" to loading).++**It does nothing for DataDome**, which ignores the claim and scores behaviour. So when someone asks+"are you still not matching Chrome's user-agent?", the answer is measured, not assumed: read the+three values above off the live window first. On 2026-08-20 they were already correct and the wall+was DataDome — a different vendor entirely.++**Same browser, same profile, same minute** (the contrast that proves it is site-specific, not a+pup-wide identity problem):++| Site | Content | DataDome |+|---|---|---|+| TI | 3,953 chars, loaded | no |+| DigiKey | 12,482 chars, loaded | no |+| **Mouser** | **0 chars** | **yes** |++## Step 3 — the sourcing ladder (most walls are avoidable, not winnable)++**1. Official API first, always.** Mouser and DigiKey both have real APIs, and Adom already wraps+them: the `adom-mouser`, `adom-digikey` and `adom-parts-search` skills talk to shared service+containers that hold the keys. **Search, pricing, stock, part lookup and datasheet URLs must go+through those, never through a pup crawl.** Crawling a page for something the vendor hands you over+an API is how you earn a velocity ban for data you could have had cleanly.++**2. `nb` when the user's identity matters.** A site that wants a signed-in human should get the+user's real signed-in browser. That is `nb`/`nbe`'s whole job.++**3. pup for what only pup can do.** chip-fetcher's actual need — downloading CAD bundles (STEP,+`.kicad_sym`, `.kicad_mod`, Altium `.IntLib`, Fusion `.lbr`, datasheet PDFs) behind click-through+flows — is not an API call, so pup is correct there. Keep pup scoped to that.++**4. Hand the challenge to the human** when one still appears (below).++## Step 4 — the human handoff (the escape hatch that always works)++pup is **headful and on-screen by design**, so the human can simply solve it. That is not a+workaround; it is the payoff for every rule in `pup-bridge-dev` about windows being reachable.++1. Do NOT retry in a loop. A retry against a live challenge raises the score and can escalate a+   soft challenge into a hard block.+2. Flash the taskbar button (`pup_alert_window`) — signal, never a focus steal.+3. Tell the user plainly which site and what you need, and hand them the window.+4. Wait for the challenge to clear, then continue from where you were.+5. **The solve persists.** DataDome sets a clearance cookie, and pup's durable profile keeps it — so+   one human solve typically buys a long quiet period for that domain. This is the single strongest+   argument for warm profiles over throwaway ones in a crawl.++## Step 5 — do not do this to yourself (how we tripped Mouser)++The 2026-08-20 trigger was self-inflicted and is the most common way a crawl gets walled: **mouser.com+was opened twice in ~15 minutes while ti.com and digikey.com were opened simultaneously**, all from+one IP. That is a textbook velocity signal. Before blaming fingerprints, check your own access+pattern — it is usually the answer, and it is the only input you fully control.++## TODO — the roadmap to feeling like the human we actually are++Ordered by value per unit of work. None of these are CAPTCHA-solving.++1. **Detect and REPORT a wall instead of returning a blank page.** Add a `botWall` field+   (`{vendor, challenged:true, url}`) plus an `errorCode` and an `_hint` telling the caller to stop+   retrying and hand off. Today a challenged page returns `ok:true` with an empty body, so every+   caller retries blindly — which is precisely the wrong reflex. **This is the highest-value item:+   it converts a silent failure into a correct one.**+2. **A first-class handoff verb** (`pup_request_human {sessionId, reason}`): raise-or-flash, tell the+   user what is needed, poll until the page's real content appears, then return control with how+   long it waited. Makes step 4 one call instead of a hand-rolled dance every time.+3. **A politeness engine, per domain**: max concurrency 1, jittered inter-request delay, exponential+   backoff on any challenge, and a hard cap on requests/minute. chip-fetcher's whole ladder should+   run through it. Cheap, honest, and it fixes the actual cause above.+4. **Warm-profile discipline for crawls**: always reuse the durable profile so clearance cookies and+   history accumulate. A cold isolated profile per fetch looks far more like a bot than one browser+   a person has used for months.+5. **Real input for click-throughs**: prefer CDP `Input.dispatchMouseEvent` with a plausible move →+   hover → click, over instant synthetic clicks at exact coordinates. pup already has isTrusted+   input paths for `nb`; reuse the shape here.+6. **Automatic routing for known-walled domains**: keep a per-domain registry (same idea as the+   `driving-sites` recipes) recording which wall a domain runs and the preferred rail (API / nb /+   pup+handoff), so the ladder is data, not folklore.+7. **Investigate CDP observability** honestly: establish whether DataDome-class walls detect CDP+   attachment at all, and if so whether there is a SUPPORTED way to reduce the signal. If the only+   answer is spoofing, write that down and stop — an arms race we re-lose every vendor update is+   worse than a clean handoff.++## The rule to keep++Wearing the user's own Chrome identity honestly, pacing like a person, using the front door when the+vendor offers one, and handing the human their own CAPTCHA — that is a browser assistant. Forging+fingerprints to defeat a detector is a scraper wearing a costume. Adom is the first, and the first is+also what survives contact with the next vendor update.