Adom Chip Fetcher
Public Made by Adomby adom
Your whole parts library — manufacturer-grade chip CAD (symbol, footprint, 3D) one tap from your EDA tool.
Speed up adom-desktop pull_file for large binaries
Problem
pull_file currently base64-encodes the entire file in memory and embeds it inside a single JSON WebSocket message. For files > ~10 MB, the message takes 30–120 seconds to transfer over the WS relay (base64 inflates by 33%, ndjson serialization isn't pipelined, and the relay buffers the whole frame). For 75 MB files (e.g. NXP reference manuals) it routinely times out at the Docker side. Real Adom-user reports: "why is this so slow?"
Files to look at
src-tauri/src/file_handler.rs—handle_pull_files()reads bytes, base64s, packs into a singlepull_files_resultWS messagesrc-tauri/src/protocol.rs—pull_files_result()constructs the WS framesrc-tauri/src/ws_client.rs— sends the frame- CLI side:
cli/src/commands.rs—pull_file()command, currently usesdesktop_command(...)with default timeout (30 s)
Desired behavior
- Stream large files in fixed-size chunks (1 MB binary frames) over a dedicated WS message type
pull_files_chunkinstead of one giant base64 JSON payload. - Use binary WebSocket frames (no base64) —
tokio-tungstenitesupportsMessage::Binary(Vec<u8>)natively; on the Docker CLI side,tungstenitedoes too. Saves the 33% inflation + JSON parse overhead. - Send progress events so Docker can show "12/74 MB" instead of hanging silent.
- Bump CLI default timeout to a function of file size:
30 + (size_mb * 0.5)seconds, capped at 600.
Wire format proposal
// File header (text JSON)
{"type":"pull_file_start","requestId":"...","name":"foo.pdf","size":78912580,"chunks":76}
// N binary chunks, each with this 12-byte prefix:
// [u32 BE: requestId-hash][u32 BE: chunk_index][u32 BE: chunk_size]
// followed by the raw bytes (no base64, no JSON)
// Completion
{"type":"pull_file_done","requestId":"...","sha256":"..."}
CLI streams chunks to disk as they arrive; final SHA256 verified once done is received.
Backwards compat
Keep the old pull_files text-frame path working for files < 5 MB (the common case; small CAD bundles, datasheets). Add a new pull_files_streamed command for the large-file path; CLI auto-picks based on stat-reported size.
Acceptance test
# Pull a 75 MB PDF — should complete in < 15 s, not 60+
time adom-desktop pull_file '{"filePaths":["C:\\Users\\john\\Downloads\\MCXNP184M150F70RM.pdf"],"saveTo":"/tmp"}'
Target: 75 MB in 15 s (5 MB/s effective throughput minimum). Current measured: timeout at 30 s with no completion.
Why this matters
pull_file is the canonical desktop→container transfer used by chip-fetcher (NXP/TI/ST/etc. CAD bundles), datasheet harvesting, log forwarding, and screenshot pulls. Any tool that needs the user's local Downloads folder → container hits this. Sub-10MB users don't notice; anyone grabbing reference manuals (commonly 50–100 MB) does. This is on the chip-fetcher skill's "must fix" list because the skill recommends harvesting all design-help PDFs per chip — which means hitting RM/SRM > 50 MB regularly.
Bonus: HTTP fallback path
Even simpler fix if streaming WS frames is too invasive: have adom-desktop spawn a one-shot localhost HTTP server (random port, single-use signed token), respond to the pull_file request with {"http_url": "http://desktop-ip:port/<token>/<file>"}, and let Docker curl it directly over the user's LAN. WebSockets stay for control messages; bulk data goes over HTTP/1.1 chunked transfer where it belongs.
# Speed up adom-desktop pull_file for large binaries
## Problem
`pull_file` currently base64-encodes the entire file in memory and embeds it inside a single JSON WebSocket message. For files > ~10 MB, the message takes 30–120 seconds to transfer over the WS relay (base64 inflates by 33%, ndjson serialization isn't pipelined, and the relay buffers the whole frame). For 75 MB files (e.g. NXP reference manuals) it routinely times out at the Docker side. Real Adom-user reports: "why is this so slow?"
## Files to look at
- `src-tauri/src/file_handler.rs` — `handle_pull_files()` reads bytes, base64s, packs into a single `pull_files_result` WS message
- `src-tauri/src/protocol.rs` — `pull_files_result()` constructs the WS frame
- `src-tauri/src/ws_client.rs` — sends the frame
- CLI side: `cli/src/commands.rs` — `pull_file()` command, currently uses `desktop_command(...)` with default timeout (30 s)
## Desired behavior
1. **Stream large files in fixed-size chunks** (1 MB binary frames) over a dedicated WS message type `pull_files_chunk` instead of one giant base64 JSON payload.
2. **Use binary WebSocket frames** (no base64) — `tokio-tungstenite` supports `Message::Binary(Vec<u8>)` natively; on the Docker CLI side, `tungstenite` does too. Saves the 33% inflation + JSON parse overhead.
3. **Send progress events** so Docker can show "12/74 MB" instead of hanging silent.
4. **Bump CLI default timeout to a function of file size**: `30 + (size_mb * 0.5)` seconds, capped at 600.
## Wire format proposal
```
// File header (text JSON)
{"type":"pull_file_start","requestId":"...","name":"foo.pdf","size":78912580,"chunks":76}
// N binary chunks, each with this 12-byte prefix:
// [u32 BE: requestId-hash][u32 BE: chunk_index][u32 BE: chunk_size]
// followed by the raw bytes (no base64, no JSON)
// Completion
{"type":"pull_file_done","requestId":"...","sha256":"..."}
```
CLI streams chunks to disk as they arrive; final SHA256 verified once `done` is received.
## Backwards compat
Keep the old `pull_files` text-frame path working for files < 5 MB (the common case; small CAD bundles, datasheets). Add a new `pull_files_streamed` command for the large-file path; CLI auto-picks based on `stat`-reported size.
## Acceptance test
```bash
# Pull a 75 MB PDF — should complete in < 15 s, not 60+
time adom-desktop pull_file '{"filePaths":["C:\\Users\\john\\Downloads\\MCXNP184M150F70RM.pdf"],"saveTo":"/tmp"}'
```
Target: 75 MB in 15 s (5 MB/s effective throughput minimum). Current measured: timeout at 30 s with no completion.
## Why this matters
`pull_file` is the canonical desktop→container transfer used by chip-fetcher (NXP/TI/ST/etc. CAD bundles), datasheet harvesting, log forwarding, and screenshot pulls. Any tool that needs the user's local Downloads folder → container hits this. Sub-10MB users don't notice; anyone grabbing reference manuals (commonly 50–100 MB) does. This is on the chip-fetcher skill's "must fix" list because the skill recommends harvesting all design-help PDFs per chip — which means hitting RM/SRM > 50 MB regularly.
## Bonus: HTTP fallback path
Even simpler fix if streaming WS frames is too invasive: have adom-desktop spawn a one-shot localhost HTTP server (random port, single-use signed token), respond to the pull_file request with `{"http_url": "http://desktop-ip:port/<token>/<file>"}`, and let Docker `curl` it directly over the user's LAN. WebSockets stay for control messages; bulk data goes over HTTP/1.1 chunked transfer where it belongs.