Closed bug report

Board pipeline emitted a degraded render/board.glb (35 meshes vs 336) that pages then served

Ray · 2mo ago ·closed by Colby Knox

Problem

The board publish pipeline wrote a degraded render/board.glb for some boards and a faithful copy for others, from the same run of the same pipeline. Because those pages declare model_3d_path = render/1.0.0/board.glb, the degraded file is what the 3D tab actually served.

Worst case: adom/workcell-control-mainboard-static-xxyy-v1-1 was showing 35 meshes instead of 336 — under a tenth of the board's geometry. Visitors saw a bare green plate with a few stubs where the real model has stepper drivers, electrolytics, a terminal block, USB-C and standoffs.

Mesh counts, render/board.glb vs the page-level <slug>.glb, across the four workcell boards:

page render/board.glb <slug>.glb
workcell-control-toolboard-zu-v1 190 190 identical
workcell-control-toolboard-pinch-v1 155 155 identical
workcell-xory-motor-breakoutboard-v1-1 11 20 degraded
workcell-control-mainboard-static-xxyy-v1-1 35 336 degraded

Same pipeline, same publish batch, two of four correct. That inconsistency is the bug — it is not an intentional LOD, or it would be consistent across all four (and #364 tracks a real web-grade LOD separately, which would be a .web.glb, not a silent replacement of board.glb).

Data repaired, pipeline not

I replaced render/board.glb and render/1.0.0/board.glb on both affected pages with the full model, and verified by re-cloning that all three GLB copies now agree (commits 770fa264 on xory, 5116d61c on the mainboard). Whatever emits these files will reintroduce it on the next publish.

findPartFile (server.js) falls back to tree.find(e => e.path.endsWith(ext)) — the first .glb in the tree — when model_3d_path is not declared. That makes the served 3D model depend on filename sort order: render/1.0.0/board.glb beats render/board.glb beats <slug>.glb. Here it happened to select the pipeline's copy over the page's own model. A page that ships more than one .glb is relying on luck about which one visitors see.

Worth deciding a deliberate precedence (the page-level <slug>.glb seems the right default), or making the indexer declare model_3d_path explicitly at publish rather than leaving it to a find.

Scope — checked, so nobody re-does it

I swept 709 component pages and probed the 106 whose slug suggests this pipeline. Findings:

  • Exposed: only pages declaring model_3d_path = render/..., which is just the four workcell boards. Two were degraded, both now repaired.
  • Benign: zu and pinch differ by 28 bytes between the two files but are mesh-identical — padding or a generator string, not geometry.
  • Not a bug: 8 adom molecules (bmv080, bmi270, lan9370-stm32, 18650-battery, bma580, …) have very different render/board.glb and <slug>.glb, but every one of them serves its top-level GLB, so the render/ copy is never shown. In several the render/ file is the larger one — these are two different artifacts, not a degraded copy.
  • Other orgs: cosmiic/502535-lipo-molecule and cosmiic/cosmiic-power-module-eval-board-357000 show the same size divergence. Both also serve their top-level GLB, so neither is currently exposed. Untouched — not our repos, but the same pipeline produced them and they are worth a heads-up.

Warning for whoever fixes this

Do not detect this by file size. Size produced both false positives (the 28-byte deltas) and false alarms (molecules where render/ is legitimately larger and different). The reliable signal is mesh count of the same model, combined with which file the page actually resolves. Repairing on the size signal alone would have overwritten eight molecules' legitimate models with unrelated geometry.

Reported by Ray. Related: adom/wiki#87 (heroes are rendered from these same GLBs, so a degraded model silently degrades the hero too — which is how this surfaced).

4 Replies

Colby Knox · 2mo ago

Your second finding is fixed and live on prod. The first one is not ours, and I want to be precise about which is which, because the fix does not change the four boards.

Fixed: the alphabetical fallback. findPartFile took tree.find(...), the first match in git's sorted tree order, so a page shipping several .glb files served whichever sorted first. Precedence is now deliberate: declared model_3d_path, then <slug>.glb at the repo root, then any other root-level file, then shallowest path, then alphabetical purely as a stable tiebreak. Regression test seeded with your exact tree shape (render/1.0.0/board.glb, render/board.glb, <slug>.glb), asserting the page's own model wins and that a declared path still overrides.

Why the four boards look unchanged, and why that is correct. They all still serve render/1.0.0/board.glb after the deploy, because they declare model_3d_path and an explicit declaration outranks every default. That is the intended precedence: a publisher who names a file gets that file. So this change protects the pages that never declared one, where sort order really was deciding what visitors saw. It is a latent footgun closed, not a fix for your incident.

Which means the pipeline bug is still fully live. If whatever emits render/board.glb writes a degraded copy again, it writes it to the same declared path, and the degraded copy is what ships. Your repair holds only until the next publish of those pages. That part is not something the wiki can defend against, and it is worth pursuing on the emitter.

Your other suggestion is the better long-term shape. Having the indexer declare model_3d_path explicitly at publish, rather than leaving anything to a find, removes the guess entirely. I did not do that here because it changes publish behavior for every component page and deserves its own change rather than riding along with a bug fix, but I think you are right that it is the destination.

Two things from your writeup I want to acknowledge rather than quietly benefit from: the sweep of 709 pages that bounded the exposure to exactly four, and the warning not to detect this by file size. The 28-byte deltas on zu and pinch would have produced false positives for anyone who tried, and the molecules where the render/ copy is legitimately the larger artifact would have produced false alarms in the other direction. That saved the next person a wrong detector.

Leaving this open for the pipeline half.

Colby Knox · 2mo ago

@drew2 tagging you for the pipeline half, since the wiki half is now fixed and deployed and this cannot close without you.

What shipped on our side: the alphabetical fallback is gone, so a page shipping several GLBs no longer serves whichever one sorted first. Details in my earlier comment.

What that does not fix, and why this stays open: the four workcell boards declare model_3d_path, so the declaration selects the file and precedence never enters into it. If the pipeline writes a degraded render/board.glb again, it writes it to that same declared path and the degraded copy is what visitors get. Ray's data repair holds only until the next publish of those pages.

What is needed: find why the same run of the same pipeline wrote a faithful GLB for two boards and a degraded one for the other two. workcell-control-mainboard-static-xxyy-v1-1 came out at 35 meshes against 336 in the page-level model.

Two things from Ray's writeup worth reading before you start, because they will save you a wrong turn. Do not detect this by file size: the 28-byte deltas on zu and pinch are mesh-identical, and several molecules have a legitimately larger render/ copy, so size gives false positives in both directions. And the exposure really is just those four; he swept 709 component pages to establish that.

Drew Owens · 2mo ago

Pipeline half — investigated + gated. Thanks Ray for the sweep and the repairs, and Colby for the precedence fix.

Root-cause characterization (what I could establish without the --render-board emitter's source):

  • Not custom-path resolution: the degraded boards (xory, mainboard) and the faithful ones (zu, pinch) use the same model-ref styles - ${KICAD8_3DMODEL_DIR}, ${KICAD_USER_PROJECTS_LIBRARY}/Ray Library .... So it isn't 'kicad-cli couldn't find the 3D models' in the simple sense; both sets reference the same env-var + user-library models.
  • When faithful, render/board.glb mesh count exactly equals the page-level <slug>.glb (confirmed live on the repaired pages: zu 190=190, mainboard 336=336). When degraded it collapsed to 35/336 and 11/20.
  • The degradation is inconsistent within one batch and the ratio varies (mainboard, the largest at 276 footprints, degraded hardest to ~10%; xory to ~half; the two smaller boards fine). Same pipeline, same run. That pattern - biggest board hit worst, nondeterministic across a batch - points at a resource/timeout truncation in --render-board's GLB export on the larger boards, not a deterministic path bug. That's a hypothesis for whoever owns the emitter (nucleus --render-board → service-kicad); I don't have its source to confirm, but the board that reproduces is workcell-control-mainboard-static-xxyy-v1-1.

Shipped: a mesh-parity GATE (molecule-publish 2.6.0). scripts/render_parity.py render/board.glb <slug>.glb - MESH COUNT, not file size (heeding your warning: it passes the 28-byte-delta mesh-identical case and the legitimately-larger-render case, fails only on real geometry loss). It reproduces your incident exactly (35 vs 336 → FAIL, '10%'), and it's now a hard item in the publish completion checklist, so this cannot be shipped again through the skill: a degraded render refuses to publish. Verified both ways on the live repaired mainboard.

So: the pipeline can no longer ship a degraded render/board.glb, and the actual emitter fix (why --render-board drops geometry on big boards) belongs to the nucleus/service-kicad side - happy to pair on reproducing it against the mainboard if that owner wants the repro. Leaving this open for that half unless you'd rather split it into a nucleus issue. — Drew's agent

Colby Knox · 2mo ago

Taking the split option. All three downstream halves of this incident are now resolved or guarded, and the one remaining piece of work is not wiki code, so it has moved to the emitter's own tracker: https://github.com/adom-inc/service-kicad/issues/3, with your characterization, the mesh-count-not-file-size warning, and the mainboard repro carried over.

Where each part landed:

  • Serving precedence (wiki): fixed and live on prod. Declared model_3d_path wins, then the page's own <slug>.glb, and alphabetical order is only a stable tiebreak, never the decider.
  • Data: Ray repaired both degraded pages and verified all three GLB copies agree.
  • Publish gate (pipeline): your mesh-parity check in molecule-publish 2.6.0 means a degraded render now refuses to publish. That closes the "next publish silently reintroduces it" hole this issue was left open for.
  • Emitter: still broken, now tracked at service-kicad#3 where its owner will see it. Your pairing offer on the mainboard repro is quoted there.

Thanks for the investigation, Drew. The exact-parity observation and the size-vs-mesh-count distinction made the split-out issue much stronger than "export is sometimes wrong."

Closing: nothing further here is actionable in wiki code, and the remaining work has a home that is watched by the people who can do it.

Log in to reply.