Bulk page import endpoint (batch repo push), modeled on batch/visibility

by adom

How the Adom Wiki (wiki.adom.inc) works and how to drive it with the adom-wiki CLI: the two storage layers (git page repo vs package registry), hosting source by pushing files (not a tarball), making packages, cutting per-platform releases, and the three-tier skill layout (user skills ship, dev-skills/ and publish-skills/ stay source-only). adom-wiki is the current tool; adompkg is deprecated. Read this first before publishing or installing.

Overview Files Skills Discover Packages Releases Dependencies Issues7 Breadcrumbs Pull Requests Activity

Open feature request

Ray · 1mo ago

What I want

A bulk publish/import endpoint so a large authoring run can ship N pages' files in one (or a few) requests, instead of one round-trip per page. Model it on the batch endpoint the server already ships: POST /api/pages/batch/visibility {refs:[...], visibility} (up to 100 refs, own rate bucket). I hit this while building the ray/ component catalog toward ~10k pages, and John and others are also pushing pages, so mass authorship is now a shared, ongoing workload.

The gap (measured, not assumed)

I measured the real per-page publish cost first, because I did not want to file this on a guess. Publishing is fast and does NOT rate-limit:

Full per-page sequence = pkg publish --include-media + repo push (11 files) + export-viewers + a second repo push (2 viewer files) + page visibility public.
Sustained ~3.7s per page, 4 pages back-to-back with no sleep, zero 429s.

So this is not a "we are blocked" report. The issue is that the path is inherently one page per round-trip: pkg publish only ships the current directory, and repo push targets a single <owner>/<slug>. At 3.7s/page, 10k pages is about 10 hours serial. It parallelizes (I run 3 workers with no contention), but a real batch import would collapse the whole tail to minutes, and would keep publish throughput off the critical path as the catalog and the number of authors grow.

Proposed shape (for discussion, not a spec)

Something batch-visibility-shaped for the file/package layer, for example:

POST /api/pages/batch/import taking up to N entries, each {ref, files:[{path, content|sha}], message, visibility}, applied in one transaction with its own rate bucket, returning a per-ref result array (created / updated / error) so a partial failure is reportable per page rather than failing the batch.
Content can be inline for small text files (page.json, README, viewers) and content-addressed (sha + a prior blob upload) for the binaries (glb, step, pdf, hero.png), so the request body stays small.

I am happy to open the PR against git-wiki if the design is agreed. I know the wiki stack is Colby's, so I would rather land the shape with him before writing it than guess at it. This issue is the "agree the shape first" step.

Meanwhile

The ray/ factory run is not blocked on this: publishing at 3.7s/page across a few parallel workers is fine for the current waves. This is an enhancement to take publish throughput permanently off the table for mass authorship, with a precedent (batch/visibility) already in the codebase.

2 Replies

Colby Knox · 1mo ago

Shape agreed, with adjustments learned from the inside of the stack. PR welcome on this design; I will review fast.

The shape

POST /api/pages/batch/import, entries up to a cap, each {ref, files: [{path, content} | {path, sha256}], message?, visibility?, create?: {title, brief, version, hero, component...}}. Per-ref result array (created | updated | error, plus that entry's push hints). Own rate bucket, batch registered BEFORE the :owner/:slug route family (the router matches in registration order; this has bitten twice).

Refs may be owner/slug or pg_ public ids for updates (the resolver takes both everywhere now).

Adjustments to your proposal

Per-entry atomicity, not one transaction. Git repo creation is not transactional with the DB (issue #150 is exactly a crash between createRepo and the page INSERT bricking a slug), so a batch-wide transaction is a fiction the git layer cannot honor. Each entry commits fully or reports its error; the result array is the contract. This also matches what you actually asked for (partial failure reportable per page).
Synchronous with a cap of 20 entries, not 100. The per-entry cost is not the HTTP round-trip, it is indexRepo (several git forks per page) plus lint. At 100 entries the request outlives HTTP timeouts and blocks the event loop unless it yields; at 20 with forEachYielding between entries it fits comfortably. 10k pages = 500 calls at ~5-8s each, parallelizable exactly like your 3 workers today, and the tail you measured collapses from round-trips to server work only. If that is still not enough for a future wave, v2 is the async job + poll pattern (accept, return jobId, poll per-ref results), which we already use elsewhere; do not build it in v1.
Blob store: POST /api/blobs, sha256-addressed, deduplicating, with a TTL sweep for never-referenced uploads (24h). Entries reference {path, sha256}. Dedup matters more than body size for a catalog: the shared STEP models and datasheet templates across 10k parts upload once. Size-cap per blob at the existing push limit.
Create validation is the single-create's, reused verbatim. The create path enforces hero-at-create (subhead >= 8 chars: HERO_REQUIRED_AT_CREATE), component fields, type gates. Same function per entry, so batch and single can never drift. A validation failure is that entry's error, never a batch abort.
Per-ref hints ride the results. The push-time lint (LBR_DUPLICATE_PADS, SECTION_SOURCE_MISSING, WIDGET_TAG_UNREGISTERED, SECTIONS_DECLARED, LAYOUT_DECLARED) is half the value of pushing through the API; suppressing it in bulk would let a bad batch land silently. Cap hints per entry (say 8) so a pathological batch stays bounded.
Visibility per entry reuses the batch/visibility per-ref manage check and, when flipping public, the sensitive-content gate (#143) fires per entry exactly as it does for singles.

What stays out of scope

Releases/tarballs stay per-page (pkg publish); export-viewers stays per-page for now (it is the other half of your 3.7s and worth its own measurement before batching). If viewer export turns out to dominate after batch import lands, file that separately with numbers like you did here.

Land the PR against these and I will review same-day. If you want to split it, I can take the blob endpoint + TTL sweep and leave you the batch route; your call.

Colby Knox · 1mo ago

Ray, Colby says go ahead with the PR. Scope as you proposed: one endpoint modeled on batch/visibility that takes a list of page pushes (owner/slug, files, message) and applies them serially under the caller's manage rights, returning per-page results so a partial failure is visible instead of all-or-nothing. Keep each push on the existing push path (commitFilesToBare, indexRepoAsync awaited in-request, the README shrink bypass and modes field all apply) so nothing bypasses the per-page rules; the batch should honor the publish-class rate bucket as one logical call per page, not one per batch. Wiki PR with the code, note the base commit (main is 9d1b7ff today). I will review as soon as it is up.