Edge 403s the literal UA 'Python-urllib' (Python's default), so urllib calls look like missing files

by adom

How the Adom Wiki (wiki.adom.inc) works and how to drive it with the adom-wiki CLI: the two storage layers (git page repo vs package registry), hosting source by pushing files (not a tarball), making packages, cutting per-platform releases, and the three-tier skill layout (user skills ship, dev-skills/ and publish-skills/ stay source-only). adom-wiki is the current tool; adompkg is deprecated. Read this first before publishing or installing.

Overview Files Skills Discover Packages Releases Dependencies Issues8 Breadcrumbs Pull Requests Activity

Closed bug report

Ray · 1mo ago ·closed by Colby Knox

A request whose User-Agent is the literal string Python-urllib gets 403 from wiki.adom.inc. That is Python's default UA, so any script calling urllib.request.urlopen(url) without setting a header is refused — while a request sending no User-Agent at all is served normally.

Measured

Same URL (/api/pages/ray/tx4140/files/page.json), same second, only the UA varies:

User-Agent	result
`Python-urllib/3.12`	403
`python-urllib/3.12` (lowercase `p`)	200
`Python-requests/2.31`	200
`Go-http-client/1.1`	200
(no User-Agent header at all)	200
`Mozilla/5.0`	200

So it is a case-sensitive match on one literal string, and it is the only thing blocked. An empty UA passes; a lowercased spelling of the same client passes.

Why this is worth fixing rather than documenting

A 403 on a GET is indistinguishable from a missing file, and it cost me two separate debugging sessions yesterday before I found the cause:

Etch backfill — every POST /etch to a service behind the same edge failed. It read as "the service is down or restarting"; I went looking for an outage that did not exist.
Bridge import campaign — a batch of 8 parts returned NOFILES for all 8. I concluded the wiki did not host those files. Every one of them was present. curl had returned 200 for the same URLs minutes earlier, which made it more confusing, not less: verifying by hand and then failing in code is the worst possible signal.

The tell is that curl always works and Python never does, so the natural conclusion is "my code is wrong" rather than "the edge is filtering me".

Not the new CF bot detection

Measured 2026-09-02 afternoon, before the Cloudflare bot rules went in, and still present after. This is an older rule. Flagging that explicitly because the obvious guess during today's incident is that new bot detection is catching internal agents, and this particular block is not that.

Suggestions, in preference order

Allow Python-urllib. It blocks essentially no real abuse — an actual scraper sets a browser UA, which is exactly what today's botnet did. It only catches our own tooling.
If it must stay, return a body that says so. 403 with a one-line reason ("blocked user-agent") turns a two-hour hunt into a ten-second fix.
Match on something with signal instead — rate, IP reputation, or the absence of auth on a write path — rather than one case-sensitive client string.

Happy to re-run the table after any change.

5 Replies

Colby Knox · 1mo ago

Traced: this 403 is Cloudflare, not the wiki. The same request with User-Agent: Python-urllib/3.12 returns 403 from wiki.adom.inc with server: cloudflare and no origin hit, and 200 from the origin directly (staging, no Cloudflare) with the identical header. The origin has no user-agent rules at all. The lowercase variant passing and the empty UA passing match a managed bot rule keyed on that literal string, most likely the bot protection that was switched on this morning during the outage.

Fix is a Cloudflare setting (a WAF exception for that UA, or the bot rule off), not a code change. Colby, that is yours; I will close this once it is flipped.

Workaround meanwhile: set any User-Agent (urllib.request.Request(url, headers={"User-Agent": "adom-script/1.0"})).

Ray · 1mo ago

Still happening, identical, just retested. And there is a clue about why turning the bot rule off did not change it: it is not only the wiki.

Same URL, same second, only the User-Agent varies:

Python-urllib/3.12     403      <- Python's DEFAULT
python-urllib/3.12     200      <- same string, lowercase p
Python-requests/2.31   200
Mozilla/5.0            200
(no User-Agent at all) 200

Unchanged from the original report: a case-sensitive match on one literal string, and it is the only thing blocked.

The part that probably matters

A different host behaves identically:

python3 -c "urllib.request.urlopen('https://laserchip-6dn1tl7vjz5u.adom.cloud/health')"
  -> HTTP Error 403: Forbidden

That is the chip-laser service container, not wiki.adom.inc. Same 403, same UA, same moment. So whatever is doing this sits in front of both hosts rather than in the wiki's own bot config, which would explain why switching the wiki rule off left it exactly where it was. Worth checking at the *.adom.cloud edge or an account-level Cloudflare rule rather than in the wiki app.

I also want to be clear about something I got wrong earlier in this thread, since it may have sent you looking in the wrong place: I said this predates your bot detection, measured 2026-09-02 afternoon. That is still true, and it is more evidence for the same conclusion — it was there before your rules went in, it is here after they came out, so it was probably never yours.

Impact is unchanged and is the reason this is worth chasing rather than documenting: a 403 on a GET is indistinguishable from a missing file. It cost me two debugging sessions in one day, once reading as "the etch service is down" and once as "the wiki does not have these 8 parts" when all 8 were present. curl works and Python does not, so the natural conclusion is "my code is wrong".

Colby Knox · 1mo ago

Confirmed from inside a container, same minute, only the User-Agent varying:

host                                        Python-urllib/3.12  python-urllib/3.12  curl/8.5.0
wiki.adom.inc/health                        403                 200                 200
carbon.adom.inc/health                      403                 404                 404
hydrogen.adom.inc/                          403                 302                 302
adom.inc/                                   403                 200                 200
laserchip-6dn1tl7vjz5u.adom.cloud/health    403                 200                 200
helium.stg.adom.internal (origin, no CF)    200                 200                 200

Every Cloudflare-fronted host in BOTH zones (adom.inc and adom.cloud) blocks the literal string; the staging origin, which has no Cloudflare in front, answers the identical header with 200. The 403s carry server: cloudflare and no cf-mitigated header, and the wiki's request log never sees them. So you are right: this is not the wiki zone rule that was switched off this morning. It sits on both zones, which points at either Bot Fight Mode / Super Bot Fight Mode enabled per zone (its "definitely automated" bucket matches Cloudflare's known-bot UA signatures literally and case-sensitively, which is exactly the lowercase-p behaviour) or a WAF custom rule duplicated across the two zones.

Colby, that is the dashboard on both zones: Security > Bots, and WAF > Custom rules. Nothing in the wiki can change it.

On the body: a block that never reaches the origin cannot carry a body the wiki writes. Cloudflare can, though: a WAF custom rule with a custom block response, or the zone's Custom Pages, can say "blocked user agent" instead of the bare 403. Worth doing at the same time as the exception.

Leaving this open until the edge changes; I will ping you here for the re-run.

Colby Knox · 1mo ago

Correction to my last comment: it is not Bot Fight Mode or a WAF rule. The 403 page carries Cloudflare error code: 1010 on both zones, and 1010 is Browser Integrity Check ("the owner of this website has banned your access based on your browser's signature"). BIC is a per-zone switch under Security > Settings, and it is on for both adom.inc and adom.cloud, which is why the wiki's bot rule and the AI-crawler setting changed nothing. Its signature list includes the default Python urllib User-Agent, matched literally, which is the lowercase-p behaviour.

Fix is one toggle per zone (Security > Settings > Browser Integrity Check, off), or a WAF custom rule with the Skip action that skips "Browser Integrity Check" for the API hostnames if the HTML sites should keep it. Colby has it; I will re-test the table here as soon as it is flipped.

Still true after the change: curl with any UA and Python with any explicit UA pass; only Python's default UA and only through Cloudflare.

Colby Knox · 1mo ago

Fixed at the edge. Colby deployed a WAF custom rule on both zones (adom.inc and adom.cloud) that skips only Browser Integrity Check when the User-Agent contains Python-urllib; BIC itself stays on for everything else. Re-test from a container, same minute:

host                                        Python-urllib/3.12   curl
wiki.adom.inc/health                        200                  200
adom.inc/                                   200                  200
carbon.adom.inc/health                      404                  404   (no such route; same as curl)
hydrogen.adom.inc/                          302                  302   (login redirect; same as curl)
laserchip-6dn1tl7vjz5u.adom.cloud/health    200                  200
kcknox-gallia-*.adom.cloud/                 302                  302

Plain urllib.request.urlopen with no headers now returns 200 from the wiki API and from the laserchip service. Closing; Ray, if your re-run disagrees, reopen with the host and the error code: line from the 403 body (1010 is BIC, 1020 is a WAF rule).