Files tab has no way to download the whole repo

by adom

How the Adom Wiki (wiki.adom.inc) works and how to drive it with the adom-wiki CLI: the two storage layers (git page repo vs package registry), hosting source by pushing files (not a tarball), making packages, cutting per-platform releases, and the three-tier skill layout (user skills ship, dev-skills/ and publish-skills/ stay source-only). adom-wiki is the current tool; adompkg is deprecated. Read this first before publishing or installing.

Overview Files Skills Discover Packages Releases Dependencies Issues8 Breadcrumbs Pull Requests Activity

Closed feature request

Ray · 2mo ago ·closed by Colby Knox

Problem

The Files tab lets you browse and fetch files one at a time. There is no way to take a whole project locally from the browser — and for most pages one file at a time is useless, because the files are only meaningful together.

Concrete case that surfaced this: a user browsing adom/probing-workcell-load-cell-board (private) on a laptop with no Adom desktop apps installed, wanting the board source. That page is 30 files / 46 MB — the KiCad project, four schematic sheets, an 8.9 MB STEP, a 7.3 MB GLB, .3dshapes/ vendor models, the BOM. Downloading it by hand is 30 clicks, and the binaries do not come down cleanly that way.

What does not work today

I probed the obvious endpoints; every one 404s:

/api/pages/<owner>/<slug>/archive
/api/pages/<owner>/<slug>/repo/archive
/api/pages/<owner>/<slug>/tarball
/api/pages/<owner>/<slug>/zip
/api/pages/<owner>/<slug>/repo/zip
/api/pages/<owner>/<slug>/download

The existing paths and why each falls short:

route	limitation
`adom-wiki repo clone <owner>/<slug>`	works and is the right answer for a terminal — but requires installing the CLI, so it is not an answer for someone in a browser
`GET /api/pages/.../files/<path>`	one file per request
`adom-wiki release download`	returns nothing unless release assets were uploaded; the package `.tgz` is a separate thing reachable only via `pkg install`
`GET /api/packages/.../:version/source`	closest existing thing, but it is version/release-scoped (needs a `releases` row), returns `.tgz` rather than zip, and is buffered — see below

Proposal

A whole-repo archive endpoint plus a Download ZIP button on the Files tab, next to Propose a change. Every other git host has this.

Two constraints that matter for the implementation:

It must ride the source gate (ensureCanReadSource), not the page gate. A public page with source_visibility = private must not hand out its whole tree as a zip — the same reasoning /api/packages/.../source already documents.
It must not be buffered. archiveRef* goes through gitBufAsync, whose 50 MB maxBuffer overflow is swallowed by a bare catch into null and surfaces as a misleading 404. Repos carrying CAD clear 50 MB on their own, so a buffered implementation would fail exactly the pages that need this most.

Fixed by #271.

A fix is up as a PR on the code repo: https://github.com/adom-inc/git-wiki/pull/271 (streamed git archive --format=zip, source-gated, + a Download ZIP button on the Files tab).

1 Reply

Colby Knox · 2mo ago

Shipped and live on wiki.adom.inc. Your PR #271 merged essentially as written: the diagnosis was right, the streamed git archive --format=zip was the right call, and the reasoning about the buffered helper's 50 MB ceiling turning into a misleading 404 saved the exact pages this exists for. Thanks for filing it with the fix attached.

Two things added on top before it went out:

Rate limit, 10/min. One request is the entire repo, which makes this the heaviest read on the site, so it is capped tighter than the per-file routes. An unthrottled loop over a public board page was the cheapest way to burn CPU and egress.
Cache-Control: private, no-store. The archive can carry a private or source-sealed repo and every request reaches us through one shared CDN, so no intermediary may keep a copy. The gate that authorized a given caller does not travel with the URL.

Verified on prod against a real repo: valid zip, correct Content-Disposition filename, private, no-store present, a bad ref returning a clean REF_NOT_FOUND rather than a truncated download, and REPO_NOT_FOUND for a tarball-only page.

The CLI picked this up the same day: adom-wiki repo download shipped in 1.0.75, streamed to disk with all four error codes mapped and 429 back-off. repo clone stays as-is for the working-copy flow.