Page renders hang site-wide (20 s timeouts) while /health, the API and static assets answer instantly

by adom

How the Adom Wiki (wiki.adom.inc) works and how to drive it with the adom-wiki CLI: the two storage layers (git page repo vs package registry), hosting source by pushing files (not a tarball), making packages, cutting per-platform releases, and the three-tier skill layout (user skills ship, dev-skills/ and publish-skills/ stay source-only). adom-wiki is the current tool; adompkg is deprecated. Read this first before publishing or installing.

Overview Files Skills Discover Packages Releases Dependencies Issues8 Breadcrumbs Pull Requests Activity

Closed bug report

John Lauer · 1d ago ·edited ·closed by Colby Knox

Symptom. HTML page renders hang site-wide while the rest of the server is healthy. Readers see pages that never load (seen in Chrome and in several pup windows), then some come back.

Measured 2026-10-19, about 6:50 to 7:00 AM Central, anonymous curl with a 15 to 20 s cap:

200 0.15s  /health
200 0.19s  /
000 15.0s  /adom/wiki             (timed out)
000 15.0s  /adom/in-s63btg        (timed out)
000 15.0s  /adom/hydrogen         (timed out)
000 15.0s  /john/buck-12v5v-molecule (timed out)
000 20.0s  /adom/widgets          (timed out, then 200 in 0.15s a few minutes later)
200 0.21s  /api/pages/adom/widgets
200 0.34s  /api/widgets
200 0.21s  /widget-script/adom-video.js
200 0.25s  /static/app.js

At about 7:00 AM Central, /adom/widgets was back to 200 in 0.15 s, while /adom/wiki still timed out at 20 s.

What it is not. Pages without any widget tag (/adom/wiki) hang the same way, so it isn't the widget registry or inline widget scripts. The API, the static assets and the widget-script route all answer in under 0.4 s. Only the server-side page render stalls. A rolling deploy or a stuck render worker would fit; worth checking what changed around 6:45 AM Central.

2 Replies

John Lauer · 1d ago

Diagnosis from adom-wiki admin health, 7:25 to 7:35 AM Central. The server is up (uptime about 17.8 h, no restart) but its event loop is choking.

Event-loop stalls of 2 to 13.5 s (big_stalls; worst 13,531 ms at 7:29 AM, gc 1.2 s). While the loop is blocked, every page render waits, which is the hang readers see.
Dozens of requests in flight for over 2 minutes each, almost all file-viewer and blob routes on adom/ component pages, spread across hundreds of different parts: GET /adom/<part>/files/docs/<part>-datasheet.pdf, …/files/<part>.glb, …-etched.step, .kicad_mod, .kicad_sym, /blob/component/…, /viewer/footprint/…. That is a crawler-style walk, not our tools. The CLI shows only a few hundred requests a day in admin clients.
The cache had just been emptied: hygiene.last_blobcache at 6:47 AM Central removed 2,914 blobs (4.06 GB of 7.49 GB), three minutes before renders began hanging. Every crawled file now reads cold from git.
Memory: RSS 12.5 GB, with external/array_buffers at 3.1 GB, which looks like whole files buffered for those responses.
The anonymous HTML memo is ineffective: 481 hits against 618,663 misses, pinned at its 50 MB cap (258,630 evictions, 194,052 deferred admits).
Listing requests are queueing: 1,598 queued, 237 refused, waits up to 23.5 s.

Likely cause. robots.txt allows every crawler on every /<owner>/<slug>/files/* page (only /api/, /viewer/ and /login are disallowed). With about 34k hardware pages carrying a dozen files each, a crawler pass over a freshly purged blob cache saturates the single Node process.

Suggested fixes, from fastest to sturdiest:

Rate-limit or challenge crawlers on /*/files/* and /blob/* at Cloudflare (bot rule or rate limit), or block the offending user agent from the access logs.
robots.txt: add Disallow: /*/files/ and Disallow: /blob/ for * (the page and its README stay crawlable), plus a Crawl-delay.
Server: stream large blobs instead of buffering them, cap concurrent cold blob reads, and don't purge the blobcache while traffic is high. The 6:47 AM sweep removed more than half of it.

Colby Knox · 1d ago

Root cause, and the fix is on prod since 13:34 UTC (0d92311).

From about 11:00 UTC, crawlers pulled thousands of different hardware part files at once (datasheets, GLB, STEP, KiCad), almost none of them read before. The catalog went from about 40k parts on 10-15 to 66.7k today, so nearly every crawled file was cold.

The server reads file contents through a pool of git cat-file processes, one per repository, capped at 40. The cap was soft: with all 40 busy, a request for another repository started one more. Big cold files made reads slow, so all 40 stayed busy, so every new repository added a process, which made every read slower still. It ended at 6,944 git processes, 13 s per read, 25,050 reads queued and 4.8 GB of file bodies held in memory. Page renders read from git too, so they hung. /health only checks the database and the disk, so it stayed green.

On the theories above:

The blobcache trim was not the trigger. It runs every hour and trims back to its 4 GB limit, and latency was already pinned from about 11:00 UTC, before the 11:47 run. That run finding 7.5 GB is the evidence instead: about 3.5 GB of cold files came through git in one hour, against 0.4 to 1 GB normally.
Not the ::adom-video change. None of the stuck requests touched widgets.
robots.txt / blocking: we're keeping everything crawlable. The fix is making these reads cheap.

What shipped: the reader cap is now hard. With every reader busy, a request waits up to 3 s for one, then falls back to the bounded git process pool, which queues and refuses on its own. A crawl like this one now gets slow or refused file responses instead of taking page renders down with it. Verified on prod: the pool peaks at 40, pages answer in under 200 ms.

Next: big part files will come from a content-addressed store on disk (keyed by blob hash, streamed, cached at the edge for good) instead of being read from git on every request, after a benchmark of gix (Rust git) against the git command line. The status signal is tracked in #231.