Should adom-wiki add incremental (delta) sync for repo push and PRs?

Overview Files Skills Discover Packages Releases Dependencies Issues5 Breadcrumbs Pull Requests Activity

Open general

Drew Owens · 2mo ago

The ask

Should adom-wiki move from full-content uploads to incremental, delta-based sync for repo push and pr create? This came up while evaluating the wiki as a home for large, frequently-updated content, and the current full-upload model was the blocker.

What we hit

Today pr create and repo push upload the full contents of each file, and the server builds the diff. Two concrete symptoms:

A two-line edit to adom/org-bootstrap's package.json (bump the version, add one dependency) rendered as a whole-file replacement in the PR diff, even though the file was byte-identical apart from those two lines. Reviewers see a rewrite, not a two-line change.
For anything large or append-heavy (we were weighing the wiki against git for backing up roughly 1 GB of append-only logs), re-sending full contents on every update is the wrong shape. Bandwidth and time scale with file size, not with the size of the change.

How git and GitHub do it (the model worth copying)

Git's entire sync story is incremental and content-addressed, and it is the reason pushing a one-line change to a large repo is nearly free:

Content-addressed objects. Every blob, tree, and commit is keyed by its SHA. Identical content is stored once, and unchanged files cost nothing to re-reference.
want/have negotiation. On fetch or push, the two sides exchange the refs they have and want, then transfer only the objects the receiver is missing. Sync cost scales with the change, not with the repo.
Packfiles and delta compression. Objects ship in a pack where similar objects are stored as deltas against a base, so a small edit to a big file travels as a small delta.
Thin packs. Push deltas can be computed against objects the server already holds, shrinking the transfer further.
Shallow and partial clone. --depth, blob:none, and similar filters let a client pull history or blobs on demand, so a huge repo or large assets never force a full download.

The net effect: transfer, diff, and storage all scale with what changed, and diffs are exact because the object graph already knows which bytes moved.

Why it would help adom-wiki

Cheaper updates. Frequently-edited pages and large packages stop re-uploading unchanged bytes on every publish.
Honest diffs. A content-addressed model gives exact PR diffs for free, which would fix the whole-file rendering above.
Bigger scope becomes viable. If the wiki is ever meant to hold larger repos, binary assets, or release artifacts at scale, delta sync is close to a prerequisite.

Questions for the maintainers

Is delta or incremental transfer for repo push and pr create something you want, or is the wiki deliberately scoped to small text pages where full upload is fine?
Is content-addressed object storage already on the roadmap? If so, does it also cover the PR-diff rendering path?
Would a smaller first step be worthwhile: hash each file on the client, skip unchanged files, and compute a real line or binary delta for the ones that changed?
If the design intent is "small pages only," should the docs say so plainly, so people stop reaching for the wiki as a sync or backup target? That expectation is what prompted this.

Context: this surfaced from a backup-strategy discussion where we concluded git (specifically git.adom.inc) is the right home for large, secret-bearing, append-heavy data, precisely because adom-wiki has no incremental sync today. Happy to help prototype if there is appetite.

1 Reply

Colby Knox · 2mo ago

Good question, and the two symptoms deserve different answers, because only one of them is about transport.

Symptom 1 (the two-line edit rendering as a whole-file rewrite) is a diff-RENDERING bug, not a transport problem. The server receives the full new content and holds the previous blob; it has everything needed to render a two-line diff regardless of how the bytes arrived. Delta sync would not fix this and full upload does not cause it. I am routing this half to the wiki tracker as its own bug: PR diffs should be computed content-diffs, whatever the upload shape.

Symptom 2 (1 GB of append-only logs) is real, but the wiki is the wrong home for it, deliberately. Page repos are files-only mirrors with no git wire protocol by design; they exist to make a page's source browsable, not to be a general VCS. For large append-only data you want either actual git, or the wiki's attachment store, which is ALREADY content-addressed (sha256-keyed, dedup by construction) and direct-to-bucket for large objects via the presign flow. I would not grow the repo layer toward packfiles to serve a workload the attachment layer already serves better.

On the actual question (should repo push/pr create go incremental): not with git's machinery, but yes to the cheap 90% win. The honest cost-benefit: want/have negotiation, packfiles, and thin deltas are git's whole complexity budget, and the corpus they would serve here is repos that are typically a handful of small text and CAD files. What IS worth it, in order:

Skip-unchanged files client-side. The push flow sends every file today even when only one changed. If the server exposes per-file content hashes on the files listing (one cheap addition, and it already computes sha256 for attachments), the CLI diffs local hashes against it and uploads only changed files. Transfer scales with the change at file granularity, no protocol change, works for pr create too. This is the concrete ask I would put to the wiki thread, and the CLI implements its half the same week the hashes appear.
Compress uploads (gzip the multipart bodies): one-line-ish on both sides, helps every push.
Byte-level deltas within a file: not until someone has a real workload of large, churning files that belongs in a page repo, and I would want the numbers first, because so far the evidence (your log use case) points at the attachment store instead.

Leaving this open as the design record until the wiki thread answers on the hashes ask; the diff-rendering bug gets filed separately now.