Page files served as 200 with an empty body while the repo has the bytes; the edge caches the empty copy

by adom

How the Adom Wiki (wiki.adom.inc) works and how to drive it with the adom-wiki CLI: the two storage layers (git page repo vs package registry), hosting source by pushing files (not a tarball), making packages, cutting per-platform releases, and the three-tier skill layout (user skills ship, dev-skills/ and publish-skills/ stay source-only). adom-wiki is the current tool; adompkg is deprecated. Read this first before publishing or installing.

Overview Files Skills Discover Packages Releases Dependencies Issues7 Breadcrumbs Pull Requests Activity

Open bug report

John Lauer · 19d ago

Problem

Files on a page are served as 200 with content-length: 0 while the page repo holds the real bytes, and the edge then caches the empty copy under the page's ?v=<commit> tag. Videos and posters on the page stop working, with no error a reader can see.

Page: https://wiki.adom.inc/john/buck-12v5v-molecule (the clips sub-readmes docs/clips.md and docs/clips-rework.md).

What was observed (2026-09-30, about 23:30 to 00:20 UTC)

Every clip 404'd for a while. GET /api/pages/john/buck-12v5v-molecule/files/docs/clips/<clip>.mp4?v=5e07f1bf637e returned 404 FILE_NOT_FOUND for files that exist in the repo at HEAD. All 9 <video> elements sat at readyState 0, networkState 2 forever. A few minutes later the same URLs served fine, with nothing changed on my side.
Nine files stayed empty. After that cleared, 9 of the page's ~145 files kept returning 200 with an empty body: 4 poster JPGs, 3 MP4 clips, docs/clips.md and README.md. The headers were right (content-type, an etag) but content-length: 0. adom-wiki repo show on the same path warned TRANSPORT_ALTERED ("the direct read ... did not match the repo's blob hash") and recovered the real bytes from the archive. So the repo is fine and the file-serving layer has an empty entry.
The empty copy outlives the fix. Re-pushing those files with changed bytes fixed the un-versioned URLs. But the rendered page kept referencing ?v=5e07f1bf637e, and the edge kept serving the cached empty bodies under that tag. The browser showed the old clips.md even after last_commit_hash and docs_indexed_commit had moved on. Only new file names (and a fresh tab) got around it.

Workaround used

Re-encode or remux each empty file (same content, new bytes) and repo push it; a byte-identical push is a no-op and does not help.
Rename the posters that stayed cached (-poster-v2.jpg) and update the markdown.

Asks

Find why the file store ends up with zero-length entries for blobs the repo has. All the affected files went up in multi-file repo push calls, some with 50+ files.
Never serve 200 with an empty body for a file whose blob is non-empty: return an error, or fall back to the repo blob the way the CLI's archive path does.
Do not let the edge cache an empty body under a ?v= tag, or purge it when the file's blob changes.
A page health check for "served size does not match the blob size" would have caught this. Today page health reports no issues for this page.

1 Reply

Colby Knox · 19d ago

Fixed and live. Cause, then each of your asks.

Cause. The server's disk filled up for a while on the evening of 2026-09-30, which matches your window. Every page file is served from a disk cache keyed by the file's blob hash. When the disk was full, writes into that cache failed, but the failure wasn't always noticed. Zero-length files were left behind and then served as the real file, which is the 200 with content-length: 0. While the disk was completely full, the cache couldn't be filled at all, which is the "every clip 404'd" phase. The repo itself was never affected, as repo show told you.

Why the file store had zero-length entries. Every cache write is now checked: the bytes on disk must match the blob's real size, or the write is thrown away. Your multi-file pushes weren't the problem; the full disk was.
Never 200 with an empty body. Every read from the cache now checks the cached size against the blob. A mismatch is deleted and refilled from the repo before anything is served, so bad copies left on disk repair themselves on their next read. If a file can't be read at all, the server now answers 503 FILE_TEMPORARILY_UNAVAILABLE with Retry-After, never cached, instead of a 404 FILE_NOT_FOUND that said the file was gone.
The empty copies at the edge. Rendered pages now give page files a new ?v= tag, so no page references the old URLs, including the ?v=5e07f1bf637e ones. The old URLs may still hold an empty copy at some Cloudflare locations, but nothing links to them anymore.
page health. It now compares each file's served size with its blob and repairs any mismatch on the spot, reported as SERVED_SIZE_MISMATCH. On john/buck-12v5v-molecule it checked 257 files and found none.

Your workarounds (the remuxed clips and the -poster-v2 names) are harmless, so keep them. If any file on the page still looks wrong to you, reply with its path. Otherwise this can be closed.