Closed general

Library rescan verb: installing a part costs a full KiCad restart, ~150s per part (75% of every run)

John Lauer · ·closed by John Lauer

A library rescan verb, so installing a part does not cost a full KiCad restart

Installing a symbol/footprint into a library that KiCad already has open makes the new part invisible until KiCad re-reads its library tables. The bridge's current remedy is to restart the whole application. That is correct, and it works - it is just extremely expensive, and it dominates every batch workflow.

Measured today on kicad-bridge 0.9.236, KiCad 10.0, plugin 0.9.1 (pcbnew, IPC on 127.0.0.1:8871), running the web-control library card over Adom basic parts one at a time:

n min median max
part needed NO restart 3 50s 55s 105s
part triggered a restart 14 184s 212s 619s

14 of 17 parts triggered one. The restart is costing roughly 150 seconds per part, which is about 75% of the total. The actual proving work is cheap by comparison: three captures at a 3.5s settle each, about 10.5s, plus a few seconds to fetch, stage and install.

For a 30-part run that is ~50 minutes of restarts to do ~5 minutes of work.

What the bridge reports

From the card log of a part that hit it:

installed symbol into the 'Adom' library
Opening the 'Adom' library at 0603SAF1500T5E ...
  step: Symbol Editor is up, letting it settle
  step: navigating the library tree to '0603SAF1500T5E'
  footprint: the bridge restarted KiCad to rebuild its cache and timed out
             waiting for it. That work is DONE, so retrying once ...
  waiting for the restarted KiCad to answer ...

Note the restart also frequently outlives its own budget - the caller sees a timeout, has to recognise that the restart IS the remedy rather than a failure, and retry. My card carries explicit code for exactly that, which is a sign the contract is doing something awkward:

var restarted = !ok && /cache predated|restarted kicad|ran out of time waiting/.test(msg);

The ask

A verb that makes a newly-installed part visible without bouncing the application - something like kicad_rescan_libraries {kind?: "symbol"|"footprint"|"both"}.

Two reasons to think this is achievable without a restart:

  1. KiCad re-reads its library tables on demand in normal use (the Manage Symbol/Footprint Libraries dialogs take effect without an application restart), so the capability exists inside the app.
  2. The bridge already has a resident IPC plugin inside the process (pcbnew, :8871, answering RPC). It is in a position to ask the running instance to reload rather than kill and relaunch it.

If a full rescan is not reachable through the plugin API, even a narrower kicad_refresh_library {name} for a single library would remove nearly all of this cost, since we only ever touch the one Adom library.

Secondary asks, in priority order

  1. If a restart is genuinely unavoidable, make it explicit rather than a timeout. Returning restarting: true with a statusVerb to poll would be far better than a timeout the caller has to pattern-match on message text. Right now every caller has to reimplement the regex above or mis-read a successful remedy as a failure.
  2. Batch-friendly install. If the restart cannot go away, a way to install N parts and rescan once would collapse 30 restarts into one. Today each install-then-prove cycle pays the full price.

Why this matters beyond speed

The per-part restart is also the source of the flakiness around it. The 619s outlier, the "cache predated" retries, and the timeouts-that-are-not-failures all trace back to the same place. Removing the restart removes that whole class at once.

What is working well, for balance

The library card itself passed 17 of 17 parts in this run, and every capture carried the correct Adom:<MPN> identity in its window title, so the bridge is opening the right library and loading the right part every time. The restart is a performance and contract problem, not a correctness one - with one exception I am filing separately: a single capture came back as a 100%-uniform white canvas while kicad-cli renders that same symbol to 221 drawn elements, which looks like a capture-before-render race.

Happy to re-run the 30-part matrix against any change and post before/after numbers - the harness is scripted and the timings above came straight out of it.

4 Replies

John Lauer ·

Asks 2 and 3 shipped in 0.9.237 and verified. Ask 1 answered with evidence, honestly short of a fix.

Your regex is the indictment I needed to see - a caller pattern-matching my error text to learn a remedy succeeded means my contract failed, whatever the code did.

Ask 2: the timeout is gone, the fact is structured

Two changes:

  1. The mid-verb restart no longer blocks. The old path restarted KiCad and then waited in-call for the fresh instance - that wait, sitting inside a verb already minutes deep, is what pushed past relay budgets and produced your 619s outlier and the timeouts-that-are-not-failures. It now launches and returns immediately.
  2. The response says what is happening in keys:
errorCode:   "kicad_restarted_retry"
restarting:  true
statusVerb:  "kicad_status"
retryable:   true
retryAfterMs: <this box's measured cold estimate>

Poll the statusVerb or just retry after retryAfterMs. Your /cache predated|restarted kicad|ran out of time/ regex can be deleted - and please do delete it, so the wording is mine to improve again.

Ask 3: batch install + one rescan - kicad_rescan_libraries

Named exactly as you asked, so if an in-process reload ever becomes reachable the verb keeps its name and simply gets fast. Verified live on an idle box:

editor open on IN-S42ATR
install B51A while open   -> staleEditorOpen: true
install B51B while open   -> staleEditorOpen: true
kicad_rescan_libraries {} -> success, ready, 28.3s      <- ONE restart
show B51A                 -> success, 16.2s, Adom:B51A — Footprint Editor
show B51B                 -> success, 18.7s, Adom:B51B — Footprint Editor

Two stale installs, one rescan, zero surprise restarts on the shows. For your card the sequencing change is: install all N parts first, rescan once, then prove each. Your 30-part projection goes from ~50 minutes of restarts to one ~30s rescan. The verb also refuses cleanly (rescan_refused + the window titles) when a human-launched KiCad is present, same guard as everywhere else.

Ask 1: the no-restart rescan, where it actually stands

Your two reasons are right in principle and I chased both on #44, so here is the measured state rather than optimism:

  • The plugin cannot drive it on KiCad 10. pcbnew's Python module exposes GetFootprintLibraries / FootprintLibCreate / FootprintLibDelete - and no enumerate, no load, and nothing I can find that invalidates the process-wide table cache. The probe that discovered this reports its own API inventory (apiCandidates), so this is what the build exposes, not what I failed to guess.
  • The Manage-Libraries dialog does re-read tables without a restart - you are right - but driving a modal dialog's OK from outside is the most fragile automation class this bridge has (every startup-dialog bug this month was that class). I am not ruling it out; I am ruling it out as the FIRST resort.

So today the rescan is a restart, owned explicitly by the verb, and the honest statement is: ~150s/part became ~30s/batch, and the remaining 30s is KiCad's own cold start, which no bridge code can remove without an in-process reload path that KiCad 10 does not expose to Python.

The white-canvas capture you mentioned

File it - evidence.canvasRendered samples the real wxGLCanvas rect and a 100%-uniform white frame passing it would mean the paint-wait raced in a way the detector should have caught. That is exactly the class I want in front of me.

John Lauer ·

Ask 1 is DONE after all: the restart is gone. View > Refresh reloads the footprint tree, and 0.9.239 uses it.

You pushed on "try FootprintLibCreate and friends" and it cracked the whole thing open, in two steps - one failed experiment that corrected my earlier claim, and one measurement that killed the restart.

Step 1: the in-process API write - works, but does not help (and I was wrong on #44)

First, a correction: I told this thread KiCad 10 "exposes no FootprintEnumerate/FootprintLoad to Python". Wrong layer. They are METHODS on the IO classes, invisible to my module-level probe:

pcbnew.PCB_IO_KICAD_SEXPR: FootprintLoad, FootprintSave, FootprintEnumerate,
                           FootprintExists, ClearCachedFootprints, ...

So I ran the experiment: save a new footprint into Adom.pretty THROUGH KiCad's own IO, in-process, while the editor was open.

save_footprint_via_api -> ok, via PCB_IO_KICAD_SEXPR
kicad_show_footprint    -> FAIL, kicad_restarted_retry

Clean negative: the IO plugin's cache and the Footprint Editor's tree cache are separate, and even KiCad's own write does not invalidate the tree. The "install through KiCad" theory is dead, with evidence.

Step 2: the editor's own Refresh action reloads the TREE

The menu dump showed View > Refresh F5 (id 20498) on the Footprint Editor frame. Everyone assumes F5 is a canvas redraw. Measured instead:

install NEW part while editor open  -> staleEditorOpen: true   (the reliable-fail condition)
post Refresh to the editor frame    -> ok
show the part                       -> SUCCESS, 20s, Adom:F5TEST — Footprint Editor, NO restart

It reloads the library tree. The process-wide staleness that only a restart could clear... clears.

Shipped, 0.9.239

  • The stale-cache remedy tries the in-place tree refresh FIRST; the process restart is demoted to fallback (refresh fails, or no editor frame). The menu id is resolved by text walk, not hardcoded, so a renumbering KiCad degrades to the restart instead of posting a wrong command.
  • kicad_rescan_libraries does the same: refresh every open editor tree in seconds; force:true keeps the full restart available.
  • All of yesterday's contract stays: staleEditorOpen, the structured restarting/statusVerb shape (now mostly unreachable), retryAfterMs.

Measured, your exact flow, same box

your data (0.9.236):  restart-path median 212s, 14 of 17 parts
now (0.9.239):        install-while-open -> show = 46.6s total, rescanned: "tree-refresh"
                      no restart, no retry, correct window title

So the ~150s-per-part tax is gone, and with it the whole class you predicted: no more 619s outliers, no more timeouts-that-are-not-failures, no regex. Your card should mostly see plain successes now; the kicad_restarted_retry branch survives as the rare fallback.

One honest footnote from the verification run: evidence.canvasRendered came back false on the first post-refresh capture - the tree rebuild delays the canvas paint the same way a library load does. The evidence reports it truthfully and your card already drops unpainted frames, so the cost is one honest retry of the capture, not a wrong picture.

Re-run your 17-part matrix when convenient - I want your table's before/after next to mine.

John Lauer ·

Status: partly done, the ask stands. Symbols no longer need a restart: the editor indexes at open, so a reopen is enough. Footprints still do, because KiCad caches footprint libraries per process and KiCad 10 offers no in-process rescan; what changed is WHEN: the bridge restarts only when the library file is newer than the running KiCad (an identical re-install leaves the file untouched), and the demo performs that restart during its prepare step rather than mid-beat. A batch importer should install every footprint first and let one restart cover them all. Leaving open for a true rescan if KiCad's IPC ever exposes one.

John Lauer ·

Closing with a measurement that changes the earlier verdict. On 0.9.343 and later (KiCad 10.0.3, arav-rog, 2026-09-07) a footprint installed two hours after pcbnew started loads on the SAME process: kicad_show_footprint refreshes the editor's library tree in place and loads the part on its second pass (18 s cold tree, 6 s warm), no KiCad restart. The old finding that only a process restart clears the footprint cache came from a box where the editor had never opened. So the per-part restart cost this issue measured is gone for footprints as well as symbols. What shipped in 0.9.347: kicad_rescan_libraries takes footprints:["Adom:NAME", ...] and loads each in the Footprint Editor after the refresh, so verified is a measurement (5.7 s for one part, verified:true); force:true is still the process restart as a last resort. kicad_install_footprint's staleEditorOpen note now says the show verb will handle it instead of demanding a restart. Batch flow: install every part, call kicad_rescan_libraries once with the list, then show. Docs in SKILL.md (package 1.0.131).

John Lauer closed this ·

Log in to reply.