# Developing chip-fetcher-lite — architecture & hard-won lessons

Read this before extending the app. Everything here is a real lesson from the build; it will save
a fresh session hours.

## Architecture in one screen

- **`server.js`** — pure-Node `http` server (no native deps). Serves the SPA + a REST API.
  In-memory `DB` persisted to `data/data.json` (debounced). Uploaded files in `data/staging/<id>/`.
- **`public/index.html`** — the entire frontend: one vanilla-JS SPA, dark Adom theme. Polls
  `GET /api/state` every 2.5s and re-renders.
- **`cli.js`** — thin wrapper (`serve`, `add`, `list`, `app`).
- No build step. Start with `node server.js` (port 7821, `CFL_PORT` to override).

### Data model (per component)
`{ id, name(=MPN once resolved), mpn, slug, type, passive, manufacturer, package, subcategory,
   pinCount, description, jlc, jlcPart, datasheetUrl, selected, footprintLocked,
   wiki:{status, slug, owner, url, variants}, _wiki:{files[], has3d},
   eda:{kicad|altium|fusion: {symbol|footprint|model3d: {present, source}}},
   uploads:[{eda,kind,name,path,...}], dest:{symbol|footprint|model3d:{name,path,exists,source,fpName}},
   names:{symbol,footprint} }`

- `eda` is recomputed (`recomputeEda`) from `_wiki.files` + `uploads`. 3D files mark `model3d` for all EDAs.
- `dest` is computed by `computeDest`/`applyDest` from the probed libraries.

## The Adom wiki (what this app reads)

- **Search:** `adompkg search <q>` (FTS). Hyphens break FTS (`chip-fetcher` → "no such column").
- **Page meta:** `GET https://wiki.adom.inc/api/v1/pages/<slug>` → `.page` (brief, owner) + `.metadata`
  (mpn, manufacturer, package, subcategory, pin_count, jlcpcb_part, datasheet_path, has_symbol/footprint/3d_model).
- **File list:** `GET /api/v1/pages/<slug>/files`.
- **Download a file:** `GET /blob/component/<slug>/<filename>` (the `/raw/` and `/api/.../raw/` paths 404).
- Components are wiki **pages** (type `component`), NOT adompkg packages — `adompkg view` won't find them.
- Most "library"/passive component pages are **stubs** (README + page.json only, no CAD). Handle gracefully.

### Description → MPN resolution
Users type descriptions ("100nF 0402 capacitor"). `classify()` resolves them to the real MPN page by
matching value+package against the FTS top component result (`titleMpn()` parses the MPN from a title
like "0402B104J160CT — 100nF MLCC Capacitor"). The component column then shows the MPN.

## The desktop (THE most important section)

**Barrett's "laptop" is Linux**, `HOME=/home/barrett-land`. Don't assume Windows.
**Always reach his file system through the `adom-desktop` bridge — never the container FS.**

His KiCad user libraries:
- `~/Documents/KiCAD/Symbols/*.kicad_sym` (JLC_*_BLT, etc.)
- `~/Documents/KiCAD/Footprints/*.pretty` (manufacturer-named: `TI_BLT.pretty`, …)
- `~/Documents/KiCAD/3D/<Manufacturer>/` (manufacturer subfolders)
- KiCad stock libs at `/usr/share/kicad/footprints/<lib>.pretty` (Capacitor_SMD, Resistor_SMD, …).

### adom-desktop gotchas (these WILL bite you)
- `desktop_list_files` is **non-recursive** and does **NOT** expand `%VARS%`/`~`. Takes a JSON arg.
- `read_file` **does** expand `~`/`%VARS%`. No approval needed.
- `shell_execute` runs `sh -c` (Linux). Needs user approval (auto-approve was on for Barrett).
- **`send_files` IGNORES its `dest` arg on Linux** — it dumps everything to `~/Downloads`. So:
  land the file, then `mv` it into place with a `shell_execute`. (`sendFiles()` does this.)
- `write_file` works but base64 on the CLI arg has a size ceiling — prefer `send_files`+`mv` for real files.

## Destination logic (computeDest)

- **Symbol** → JLC category lib by type: `JLC_<Cat>_BLT.kicad_sym` (Capacitors/Resistors/Inductors/
  LEDs/Diodes/Oscillators/FETs/Connectors/Flash/Power/MCUs/Misc). MCUs matches MPN hints (stm32/nrf/…).
- **Footprint** → prefer the wiki footprint filed into the manufacturer `.pretty`; for passives NOT on
  the wiki, use the KiCad **stock** lib (Capacitor_SMD → `C_0402_1005Metric`, name LOCKED, "KiCad default" badge).
- **3D** → manufacturer subfolder (deepest), matched to `metadata.manufacturer`.
- **`matchByMfr`** matches a manufacturer to a lib/folder by **exact name-segment** (`TI_BLT`→`ti`); fuzzy
  substring only when BOTH sides ≥4 chars; aliases (`MFR_ALIASES`) expand ONLY on a full-name match; generic
  words (`MFR_STOP`: technology/corporation/…) are filtered. Without these, "corpora**ti**on"→TI and
  "in**st**ruments"→ST false-matched. Don't loosen it.

## Filing into KiCad (send-to-library)

- Footprint/3D: download → `sendFiles()` (land + mv).
- Symbol: `read_file` the dest lib → `extractSymbolBlock` (balanced-paren) → `renameSymbol` →
  `stripExtraProps` (keep ONLY Reference/Value/Footprint/Datasheet/Description) → set Description
  (comma-separated specs), Footprint (`<nick>:<fpName>`), Datasheet, LCSC(=jlcPart) → `mergeSymbol`
  REPLACES an existing symbol of that name (re-send updates) → `.bak` backup → `sendFiles` (overwrites).
- Stubs (no CAD) are skipped, not errored.
- Symbol descriptions: passives = value/size/power(or voltage/dielectric); ICs = manufacturer, package,
  pin count, subcategory — all comma-separated into the single Description field (NOT separate fields).

## Filing pitfalls learned at scale (138-part export)

- **Symbols must merge per-library in ONE read→merge-all→write.** A per-component read-modify-write
  of the shared `.kicad_sym` races/sees stale content and leaves each lib with only the LAST symbol.
  `fileSymbolsBatched` groups components by target lib, reads once, merges every symbol, writes once.
  (Footprints/3D are independent files so they don't have this problem.)
- **A symbol's `Footprint` property nickname MUST match the fp-lib-table nickname**, or KiCad shows
  "no pads" (it can't resolve the footprint). If footprint libs are registered with a prefix (e.g.
  `TEMP_Royalohm` for a temp export), the symbol's Footprint must be `TEMP_Royalohm:<fpName>` —
  `buildSymbolBlock(c, {nickPrefix})` handles this. The `<fpName>` is the `.kicad_mod` FILENAME
  (KiCad ids footprints by filename, not the internal `(footprint "…")` name).
- **KiCad reads sym-lib-table/fp-lib-table only at LAUNCH** and may rewrite the global tables on
  exit — so edit the tables while KiCad is CLOSED, and have the user restart KiCad to see new libs.
- **Temp/dry-run export:** `fileToLibrary(comps, {tempRoot})` mirrors Symbols/Footprints/3D under a
  temp dir, skips lib-table registration, copies KiCad STOCK footprints in too (read from the share
  dir), and restores real dests after — non-destructive to the user's real libraries. Temp footprint
  libs are registered with a `TEMP_` nickname, so the symbols' Footprint property must use that prefix
  (`buildSymbolBlock(c, {nickPrefix:'TEMP_'})`) or KiCad shows "no pads".
- **Defensive footprint repair:** `normalizeFpLayers()` requotes `(layers F.Cu,F.Paste,F.Mask)` →
  `(layers "F.Cu" "F.Paste" "F.Mask")` on copy. The `adom-footprint` generator historically emitted the
  comma/unquoted form, which KiCad reads as an SMD *aperture* (no copper). Fixed at the source in
  [email protected], but kept here as belt-and-suspenders for any still-bad source footprint.

## Throughput / what limits requests

- The ONLY self-imposed throttle is `CHECK_CONCURRENCY` (6) on the mass-add wiki checks — each check is
  one `adompkg search` subprocess + 2 wiki GETs; the cap stops a 138-part add from spawning 138 procs.
- Send/file speed is bounded by the **adom-desktop bridge** (serial `send_files`→`mv` round-trips per
  file) + wiki blob download latency. Symbols are batched (one write/lib) so they're no longer the cost.
- Bulk wiki writes are bounded by the **wiki's own rate limit (HTTP 429, ~10/min)** — pace bulk commits.

## ⚠️ NEVER clobber a user's symbol library (data-loss incident)

A symbol lib is ONE file merged into via read→merge→write. If the read silently fails and the code
falls back to an empty template, the write **destroys every existing symbol** in the user's library.
This actually happened — it truncated real `JLC_Capacitors_BLT` (30→1) and `JLC_Resistors_BLT` (53→1).
`fileSymbolsBatched` now has two hard guards (keep them):
1. **Empty template ONLY when the file is genuinely absent.** If `read_file` fails but `[ -f path ]`
   says the file EXISTS, ABORT that library — never overwrite a lib we couldn't read.
2. **Never shrink.** Count symbols (`(property "Reference"`) before and after the merge; if the result
   has fewer than the file already had, ABORT — refuse the write.
Both guards add an error to `DB.send.errors` and skip the lib (no data lost). The old per-component
`fileSymbol` (which lacked these guards) was REMOVED — only the guarded batched path remains.

## Hard rules

- **Re-checking the wiki must NEVER delete user-uploaded files or user-edited fields.** `runCheck`
  preserves `uploads` and a manually-set `jlcPart` (`meta.jlcpcb_part || c.jlcPart || null`); edited names
  survive (only reset on a footprint lock-state change).
- **Dedup:** adding a duplicate name, or a description that resolves to an MPN already listed, is rejected.
- **Bounded wiki checks:** mass-add uses a 4-concurrent queue (`enqueueCheck`/`pumpChecks`) — never fire
  N `adompkg` processes at once (138-part stress test confirmed this is necessary).

## Local dev gotchas

- **`pkill -f "node server.js"` matches its OWN shell** (the start command contains that string) and
  kills itself before node starts. Kill by PID (`ps -eo pid,args | grep "[s]erver.js"`) or
  `fuser -k 7821/tcp`, in a SEPARATE command from the `node server.js` start.
- Start the server as a background task with `exec node server.js` (cwd = repo). Don't combine kill+start.
- API calls in the SPA are **relative to the page** (`new URL('api'+path, document.baseURI)`) so they work
  behind the Hydrogen proxy (`/proxy/<port>/`). Absolute `/api/...` hits the proxy origin and 404s.
- Fonts: Adom brand `@font-face` from `adom.inc/fonts/...` (verified 200). Never system fonts as primary.
- **Sticky top bar:** the header + controls + table column-header must stay pinned while rows scroll.
  Two traps: (1) `body{height:100%}` constrains sticky children to one viewport so they scroll away —
  use `body{min-height:100%}` instead; (2) the thead's sticky `top` must equal header+controls height,
  which CHANGES when the controls bar wraps — compute it in JS (`updateStickyOffsets` sets `--header-h`
  and `--stick-top`, called on render + resize), never hardcode pixels.
- The webview tab is created with `adom-cli hydrogen webview open-or-refresh --name "Chip Fetcher Lite"
  --url <proxyURL> --panel-id <pane>`; `--panel-id` is required the first time (find via
  `adom-cli hydrogen workspace tabs`). Reload picks up server-side file changes (server serves from disk).