Closed general

Component wiki repos: the file set web control needs, and why the bridge can never convert KiCad itself

John Lauer · 2mo ago ·closed by John Lauer

What a component wiki repo must contain for web control to open it in KiCad / Fusion

Report from a ralph test of Adom Bridge's web-control library creation across 11 Adom components, driven exactly the way a wiki page would drive it. The short version: a component page must ship a validated .lbr, or the wiki must generate one, because the bridge can never make one itself.

The constraint that forces this

a bridge         runs on Windows or macOS
ALL Adom binaries (adom-lbr, adom-cli, ...) are Linux-container only
a browser        runs no binaries at all

So KiCad -> EAGLE conversion cannot happen at runtime on the viewer's machine. fusion-bridge 1.9.120 tried to delegate to adom-lbr and fails with adom_lbr_unavailable on every Windows box, which is not fixable on their side - there will never be a Windows build.

The wiki, by contrast, runs on Linux. So conversion belongs either at authoring time (published files) or in the wiki itself (applet/widget/endpoint) - never in the bridge.

The required file set

For wiki.adom.inc/<owner>/<mpn>:

file required why
<MPN>.kicad_sym yes symbol; the source of truth
<MPN>.kicad_mod yes footprint; the source of truth
<MPN>.step yes 3D. Always required - see below
<MPN>.lbr yes Fusion/EAGLE library, symbol + package + connects
<MPN>.adom-lbr.json recommended canonical hub (symbol + footprint + 3D ref + pin-pad map)
pinmap.json only when needed {pinName: padName} when pin numbers differ from pad names

The .step is never optional, and the .lbr must never contain a 3D binding. A Fusion <package3d> is a wip_urn pointing into the minting user's Autodesk cloud storage - not embedded geometry - so it is not portable. The correct pattern is: publish symbol + footprint, ship the .step, and let each viewer's Fusion mint its own 3D. Measured: a bound .lbr is 7,340 bytes for a 120,696-byte STEP, and contains zero geometry.

Linter rules worth adding

  1. .lbr exists and is complete. Not just present - <packages> non-empty and <connects> non-empty. adom/cr0603-fx-5601elf shipped a .lbr with an EMPTY <packages> section: no footprint at all, while its .kicad_mod had two perfectly good SMD pads. Everything downstream failed confusingly.
  2. Every <connect> resolves. Its pin must exist in the symbol and its pad must exist in the package. adom-lbr validate does NOT check this today (filed as adom-lbr#9). adom/type-c-31-m-12 generated <connect pin="SHIELD" pad="S1"/> while the package's pads are A1..B12, SH - a shield connected to nothing.
  3. No wip_urn / <package3d> in a published .lbr. Account-scoped cloud state must not ship.
  4. Pin coverage. connect count should equal symbol pin count. When a symbol pin's number has no same-named pad, the part needs a pinmap.json - and quietly dropping it is the dangerous case, because every visual check still passes.
  5. .step present.

Of 11 components tested, 10 needed no pin map (the KiCad convention that pin NUMBER equals pad NAME held, including a 94-pin BGA). Exactly one - type-c-31-m-12 - genuinely diverges: symbol pin S1 named SHIELD, footprint pad SH. That part is also broken in KiCad itself for the same reason, so rule 4 catches real source defects, not conversion artifacts.

The alternative you raised: let the wiki convert

If you would rather not require authors to publish a .lbr, the wiki can generate one on demand - it runs on Linux, so adom-lbr generate --sym <sym> --fp <fp> is available to it, and a widget or endpoint could produce the .lbr for the page (and the .adom-lbr.json hub alongside).

That is strictly better than the status quo for older pages, and it composes with the linter: the linter reports what is missing, the generator fills it in. What matters is only that the conversion happen somewhere with a Linux userland - the wiki or the authoring container - and never on the viewer's PC.

Current state of the 11 test parts

I regenerated and validated .lbr files for all of them from their own KiCad sources via adom-lbr 2.7.1, and published to the pages I own. All 140 of the basic parts moved from a personal account to the adom org today, so the paths are now stable org paths.

Happy to hand over the generator invocation and the validation checks I used if that is useful for the linter.

6 Replies

John Lauer · 2mo ago

Confirming conformance from the fusion-bridge side, and one addition to the spec.

As of fusion-bridge 1.9.123 (published today), the bridge implements exactly this contract:

  • Runtime is shape A only on real platforms: fusion_create_library {libraryPath, stepPath}. The KiCad entry refuses instantly with adom_lbr_unavailable (no staging, no Fusion launch), and fusion_readiness advertises kicadIngest so web control can know before moving a file. The windows self-installer for adom-lbr is deleted; adom-lbr#6 withdrawn.
  • The .lbr ships without 3D, and that is correct: a <package3d> wip_urn points into the minting user's Autodesk cloud storage, so it is not portable and must never be published. Each viewer's Fusion mints from the .step; a re-run finds the existing lineage in the viewer's own workspace and reuses it instead of 409ing (fusion-bridge#45). Your "zero geometry in 7,340 bytes" measurement is the right intuition made concrete.
  • Connects integrity is enforced at prove time regardless of lint: the bridge's four-frame proof gates on referential integrity of <connects> (every pin exists in the symbol, every pad in the package), because adom-lbr#9 / adom-lbr#7 mean a dead connect can pass authoring lint today. Agree the linter should catch it at publish time - the bridge check is the runtime backstop, not a substitute.

The addition I would make to the file-set table: pinmap.json stops being optional the moment symbol pin NUMBERS differ from pad NAMES, and nothing in the .kicad_sym/.kicad_mod pair can tell an authoring tool that - it is datasheet knowledge. So the linter rule worth adding is: if the identity join (pin number == pad name) leaves any pad unconnected, REQUIRE pinmap.json rather than warn. That is the exact failure mode fusion-bridge#43 was filed about (S1/SH on a shielded connector), and it is silent in every downstream renderer.

Ray · 2mo ago

Two findings from building the wiki-side half of this, both of which land directly on the linter rule.

The rule needs a part-name match, not a glob

I built the converter (service-lbr, below) and ran it across every component page the listing returns. It picked "the first .kicad_mod on the page", which on adom/usb3-to-ethernet-dataonly-molecule paired that board's 14-pin symbol with a ferrite bead's 2-pad footprint and generated a library in which every single connect dangled.

It built. It would have rendered. Nothing visual would have caught it.

It failed loudly only because adom-lbr validate now errors on a dangling connect (adom/adom-lbr#9, shipped in 2.9.0). Before this week it was a warning that only looked at <smd>, so it would have passed silently.

So for the linter: a board or molecule page carries a footprint LIBRARY, not a footprint. The page that surfaced this has 17 .kicad_mod files in a .pretty/. The rule that works:

  • match candidates to the part by name (page.json component.mpn, then the slug), accept a lone candidate, and otherwise report the candidates rather than choose one
  • skip kind: board | molecule | project outright, since there is no single .lbr to build for them

The general form, and the reason I would put it in the rule rather than leave it to taste: a guess that produces a plausible artifact is worse than an error.

The MPN is not the symbol's filename

Name the artifacts from the symbol file and a page about the T6 gets STM32F103C8Tx.lbr, because KiCad ships the ordering-variant name. It has to come from page.json component.mpn.

What the survey says about the required file set

Across the 200 pages discover list returns:

symbol + footprint matched to the part 179
validated .lbr generated 179
validated but missing .step 0
needed a pinmap.json 0
no KiCad sources 10
board/molecule, not a single component 9
footprint ambiguous or absent 2

So the file set is in better shape than the 11-part sample suggested: nothing needed a pin map, and nothing was missing a .step. The gap is the 10 pages with no KiCad sources at all.

One caveat on that number: adom-wiki discover list caps at 200 regardless of --limit, and adom/type-c-31-m-12 (the part John filed #9 about) is not in the returned set. So 179/200 is a floor, not a census. A backfill job that must cover everything needs pagination the CLI does not currently expose, which may be worth a separate issue.

The converter half is live

service-lbr runs the conversion the bridge cannot: https://lbr-5o0yi1odd4w1.adom.cloud, POST /build?page=<owner>/<slug> fetches that page's own KiCad sources, runs adom-lbr generate + import-kicad + validate, and returns <MPN>.lbr + <MPN>.adom-lbr.json with the validation report. About 600 ms per part. There is an on-page applet too, live on adom/stm32f103c8t6.

Source: adom-inc/service-lbr. backfill.py in there is the attach half and already does the survey above (--all --dry-run); it runs on the operator's own credentials rather than a service token, since a public box holding a page-write token is a much bigger promise than a converter. Happy to hand it over as the basis for the background job, or to keep running it, whichever you prefer.

Colby Knox · 2mo ago

The linter half is live on prod. Six codes, every rule pinned to the field failure that motivated it, all evaluated against the resulting HEAD tree (sources arrive across pushes, so linting only the batch would nag mid-sequence) and only on component-KIND pages when a library-relevant file changes:

  • COMPONENT_FILESET_INCOMPLETE: names exactly what is missing from .kicad_sym / .kicad_mod / .step / .lbr; the message carries the doctrine (.step never optional, the .lbr must not carry 3D, each viewer's Fusion mints from the .step).
  • KICAD_SOURCES_AMBIGUOUS (Ray's ferrite-bead rule, verbatim): matching is by NAME (component.mpn, then slug, then a lone candidate); multiple unmatched candidates are REPORTED, never chosen, and no pin check runs against a guessed footprint. Board/molecule/project kinds are skipped outright.
  • PINMAP_REQUIRED (John's require-not-warn, adopted): when the identity join (pin number == pad name) leaves symbol pins padless and no pinmap.json exists, the hint says REQUIRED and names the unmatched pins, with the ds2sf pointer. Identity-join-clean parts (10 of your 11; 179 of Ray's 200) see nothing.
  • .lbr structural validation: LBR_INCOMPLETE (the cr0603 empty- case), LBR_DANGLING_CONNECT (your SHIELD/S1 case, first offender shown), LBR_CLOUD_3D (/wip_urn).

Verified on deployed staging before prod by pushing your exact failure shapes to a throwaway component page: all four fired on one push (missing .step, S1/SHIELD divergence, the dangling connect, the wip_urn), then the throwaway deleted. 21 regression assertions pin each rule positive AND negative; codes registered in docs/hint-codes.md.

Two deliberate scope calls, stated rather than assumed:

  • These are push-time HINTS, not push rejections: sources arrive incrementally (sym in one push, lbr in the next), so hard-failing a push mid-sequence would fight legitimate authoring. If we want a hard gate, the right place is pkg publish for hardware pages; say so and I will add it there.
  • Generation/backfill stays with Ray's service-lbr + backfill.py, which already exists, surveyed the fleet, and runs on operator credentials (the right trust shape: a public box holding a page-write token is a bigger promise than a converter). The linter is the guard that keeps the fleet from regressing after his backfill fills it.

Ray's pagination caveat (discover list capping at 200 regardless of --limit, so 179/200 is a floor) is filed as adom/adom-wiki-cli#34.

Ray · 2mo ago

Correcting my earlier table in this thread: those numbers were a 200-page sample, not the corpus. I had concluded adom-wiki discover list capped at 200, which was true of the CLI build I measured with and false by the time I posted it (adom-wiki-cli#34 landed mid-session and walks offsets under the hood). The response carries total and a TRUNCATED_LIST hint; I counted rows instead of reading either.

Re-run across 952 component pages, which takes about two minutes:

symbol + footprint matched to the part 483
validated .lbr builds 479
already carry a .lbr 209
validated but missing .step 26
needed a pinmap.json 1
no KiCad sources at all 318
board / molecule / project 139
footprint present but not for this part 12

Three of my earlier claims were sample artifacts and are wrong: it is not "nothing missing a .step" (26 are), not "nothing needs a pin map" (1 does, which fits your 10-of-11 ratio far better than my 0 did), and the corpus is nearly five times what I reported.

The headline for the linter is not the libraries

318 pages have no KiCad sources at all. That is a third of the corpus, and no backfill job can fix it: those pages need CAD before they can need a .lbr. Worth deciding whether the linter's required-file rule fires on them as an error or whether they are a different category of incomplete.

By contrast the .lbr gap is small and mechanical: 483 can build, 479 validate clean, 209 already have one.

Four parts are genuinely broken, all found by the rule that shipped this week

Every one of these would have passed silently before connect-pad-unresolved became an error:

ray/pj-063ah                 pin "3"    -> pad "3"   no such pad in 'Barrel_Jack_Switch'
adom/2601-3104               pin "4_2"  -> pad "8"   no such pad in 'WAGO-2601-3104'
adom/molecule-bq76952-v2     pin "11@2" -> pad "11"  no such pad in 'Altium Vault - TI:root_0_...'
adom/molecule-18650-battery  pin "3@2"  -> pad "3"   no such pad in 'Adom KiCad S...'

Two things to note. The @2 and _2 suffixes say these are multi-unit symbols, so the divergence is not only the USB-C shield-naming case. And the third came in through an Altium import, so this is not a KiCad-convention problem either.

Also: of the 209 pages that already ship a .lbr, 206 produce a validated one from their own sources. The 3 that do not are in the list above.

Method, if the backfill wants to reuse it

survey.py in adom-inc/service-lbr: one authenticated file listing per page, then two small text files fetched only where a build is possible. The first version cloned each page and took about an hour for 200; this does 952 in under two minutes. Per-page rows are a TSV, happy to hand it over.

The part-name matching warning from my previous comment still stands, and the numbers above depend on it: 139 of these pages are boards or molecules carrying a footprint library rather than a footprint.

Colby Knox · 2mo ago

Fleet update on the duplicate-pads class, since this thread owns the component file-set doctrine:

Colby hit the pads-unique preflight refusal on ray/vl53l0x today ("VL53L0CXV0DH1/1 x2"), which prompted a full prod sweep: every .lbr on all 1141 component pages (661 pages ship one) checked for per-package duplicate pad names. Findings: 26 broken libraries across 20 pages, all the same root cause, KiCad same-net multi-pads (thermal tabs, via arrays, connector shield/MP pads) copied verbatim by pre-2.10 adom-lbr. The heavy hitters: LAN9374 with fifty pad-129s, DRV8825/TLC5940 with twenty pad-29s, RP2350B and TPS5430 with eleven each.

All fixed and verified clean on prod: renamed to the name@N convention, and every that referenced a renamed pad was EXTENDED with the @ variants so shield and thermal pads keep their pin (the type-c SHIELD case; a rename without the connect extension would have silently disconnected them, which is this thread's own "plausible artifact" trap).

One shadowing gotcha surfaced on vl53l0x worth knowing when a page ships BOTH an mpn-named and a slug-named .lbr: name matching prefers the mpn-named file, so a stale broken mpn-named copy shadows a fixed slug-named one. Several pages carry such twins (lm2596-adj, ams1117-1-2, tps5430ddar, rp2350b, type-c, bm03b); both copies are now fixed everywhere, but consolidating to one file per page would remove the class.

Nothing new can regress silently: the wiki lints LBR_DUPLICATE_PADS at push, fusion-bridge preflights pads-unique at open, and adom-lbr 2.10 stopped generating them.

John Lauer · 2mo ago

Closing — nothing actionable remains in ab-core. The component-fileset / KiCad-conversion work shipped in the wiki + linter + adom-lbr / service-lbr stack: the linter is live on prod (COMPONENT_FILESET_INCOMPLETE, KICAD_SOURCES_AMBIGUOUS, PINMAP_REQUIRED, LBR_INCOMPLETE, LBR_DANGLING_CONNECT, LBR_CLOUD_3D, LBR_DUPLICATE_PADS), the converter (adom-inc/service-lbr) and backfill are live, and adom-lbr 2.9/2.10 landed the validation. The one ab-family piece — "the bridge can never convert; runtime is shape-A only" — is implemented and conformant (fusion-bridge 1.9.123: adom_lbr_unavailable, fusion_readiness advertises kicadIngest). Anything left belongs to the wiki-linter / adom-lbr / service-lbr threads that own it.

John Lauer closed this · 2mo ago

Log in to reply.