← Commit history

Publish 0.1.5

Kyle Bergstedt ·f1f84efbaf · ·parent 4d60f52
4 files changed +49−45
README.md+35−38
@@ -5,15 +5,12 @@ The instruments at our node that generate data, the file types each produces, and the rate at which they produce it today and projected as we scale. -Our node is an electronics-prototyping cloud lab. Instruments fall into two-groups, and the distinction is what drives every number below:+Our node is an electronics-prototyping cloud lab. We plan on **about 1.2 TB per+workcell per day**: about 200 GB of video and about 1 TB of sensor data. Video is+steady; sensor data is bursty and is the highest value per byte. -- **Per-workcell instruments**: one of each per workcell. These scale linearly-  with workcell count, and they dominate our volume.-- **Bench instruments**: shared across the lab. These are bursty, not-  continuous, and they do not scale with workcell count.--**We run 20 workcells today, each with one camera, and project 1,000 by 2029.**+**We run 20 workcells today, producing about 24 TB/day. We expect 100 by the end of+2027 (about 120 TB/day, or 44 PB a year) and project 1,000 by 2029.**  --- @@ -23,7 +20,7 @@ groups, and the distinction is what drives every number below:  | Instrument | File types | Rate | |---|---|---|-| High-resolution live camera | `.mkv`, `.ts` (H.264/AVC) | records 4K (3840×2160) at 60 fps, ~50–80 Mbps; simultaneously streams 1080p or 1440p at 60 fps to users |+| High-resolution live cameras (multiple 4K60 streams per workcell) | `.mkv`, `.ts` (H.264/AVC) | records 4K (3840×2160) at 60 fps, ~50–80 Mbps; simultaneously streams 1080p or 1440p at 60 fps to users | | Microphone | `.wav` | 44.1 kHz, 176.4 kB/s | | Power sensor | `.wav` | audio-rate waveform stream | | BME690 — barometric pressure, temperature, humidity | `.csv`, `.txt`, `.bin`, `.json`, `.jsonl` | low rate, typically 1–10 Hz |@@ -41,9 +38,10 @@ record is the archived artifact. | FLIR thermal cameras | `.jpg` (radiometric) | frame rate set per experiment | | µV/µA voltage and current sensing | `.csv`, `.bin`, `.json`, `.jsonl`, `.wav` | sample-rate dependent | -These are our highest-rate instruments by interface speed, but they capture in-short bursts around a device under test rather than streaming continuously, so-their contribution to daily volume is far smaller than the camera fleet's.+These are our highest-rate instruments by interface speed. They capture in short+bursts around a device under test rather than streaming continuously. Sensor data,+led by oscilloscope captures, is most of our daily volume: about 1 TB of the 1.2 TB+per workcell.  ### Custom user sensors @@ -68,54 +66,53 @@ Per-unit, per-day estimates for each instrument. Assumes 8 active hours per day. | Microphone | ~5 GB | | BMI270 IMU | ~46 MB at 100 Hz, ~740 MB at 1.6 kHz (binary; ~10× as JSONL) | | BME690 | under 30 MB |-| **Per-workcell subtotal** | **~240 GB** |-| Oscilloscope (MXR) | burst-limited; single-GB to tens of GB typical, ~1 TB per workcell on a heavy probing day |+| Oscilloscope (MXR) | burst-limited; single-GB to tens of GB on a light day, up to ~1 TB | | Spectrum analyzer | burst-limited; tens of GB, TB-scale if streamed | | FLIR thermal | MB for single shots, up to ~40 GB continuous | | µV/µA sensing | sub-GB logging to ~70 GB at 100 kHz | | Custom user sensors | user-defined; unbounded | +We plan per workcell, not per instrument: **~200 GB/day of video and ~1 TB/day of+sensor data**. The ~1 TB is a planning figure driven by bursty oscilloscope captures,+so it sits above what the light-day values in this table add up to.+ ---  ## Current vs. projected -Per-workcell instruments dominate, so total volume tracks workcell count almost-linearly.+Raw volume tracks workcell count. -| | Today | 1,000 workcells (projected by 2029) | 10,000 workcells (beyond the 2029 projection) |-|---|---|---|---|-| Workcells | **20** | 1,000 | 10,000 |-| Cameras | **20** | 1,000 | 10,000 |-| Mics / power / BME690 / BMI270 | 20 each | 1,000 each | 10,000 each |-| Per-workcell volume | ~4.8 TB/day | ~240 TB/day | ~2.4 PB/day |-| Bench instruments | ~0.05–0.5 TB/day | ~0.05–0.5 TB/day | ~0.05–0.5 TB/day |-| **Total** | **~5 TB/day** | **~240 TB/day** | **~2.4 PB/day** |-| Sustained average | ~0.5 Gbps | ~22 Gbps | ~222 Gbps |--These are typical days. A heavy probing day can produce ~1 TB of oscilloscope data-per workcell doing the probing, on top of the bench figure above.+| | Today | End of 2027 | Projected by 2029 | Beyond 2029 |+|---|---|---|---|---|+| Workcells | **20** | **100** | 1,000 | 10,000 |+| Video (~200 GB per workcell-day) | 4 TB/day | 20 TB/day | 200 TB/day | 2 PB/day |+| Sensor data (~1 TB per workcell-day) | 20 TB/day | 100 TB/day | 1 PB/day | 10 PB/day |+| **Total** | **~24 TB/day** | **~120 TB/day** | **~1.2 PB/day** | **~12 PB/day** |+| Per year | ~8.8 PB | ~44 PB | ~438 PB | ~4.4 EB |+| Sustained average | ~2.2 Gbps | ~11 Gbps | ~111 Gbps | ~1,111 Gbps |  ### Uplink  **28 Gbps is our potential uplink once the factory is fully built out**, not what is installed today. -That number is worth holding next to the table above. A 28 Gbps uplink is ~302-TB/day if saturated, which covers roughly **1,000 workcells** — our projected-volume there is ~22 Gbps sustained, comfortably inside it. At 10,000 workcells we-would need on the order of 222 Gbps, roughly eight times the fully-built-out-figure. Somewhere past a few thousand workcells, egress becomes the binding-constraint rather than the instruments, and the question shifts from "how much-can we generate" to "what do we keep, what do we reduce at the edge, and what do-we ship."+A 28 Gbps uplink is ~302 TB/day if saturated, which covers the raw capture of+roughly **250 workcells**. Raw capture from 1,000 workcells would need about four+times that, and from 10,000 about forty times.++So raw capture does not all go out. It lands in local caching at our Cloud Lab, and+we publish derived metadata. We expect 50:1 to 100:1 compression from raw data to+derived metadata, which would put 1,000 workcells at about 12 to 24 TB/day, well+inside the uplink. The question is "what do we keep, what do we reduce at the edge,+and what do we ship."  ---  ## On these numbers  The instrument inventory, the file types, and the native interface rates are-firm. The daily volumes are **estimates**: they assume 8 active hours per day and-one of each per-workcell instrument per workcell. Actual volume depends on real+firm. The daily volumes are **estimates**: they assume 8 active hours per day, and+the 50:1 to 100:1 compression is an expectation we have yet to measure. Actual volume depends on real duty cycles, which we will have once InstaPCB goes live in early 2027. This page is versioned and will be updated in place with measured figures then. 
SKILL.md+12−5
@@ -32,9 +32,11 @@ what is installed today. All figures are estimates pending measured numbers after InstaPCB goes live in early 2027. "Active" = 8 h/day of workcell operation. +- Planning figures per workcell: ~200 GB/day of video (multiple 4K60 streams) and+  ~1 TB/day of sensor data, ~1.2 TB/day in total. - 4K60 camera: ~50–80 Mbps → ~200–290 GB/day active, ~650 GB/day continuous, per camera. - Spectrum analyzer: 5 Gbps peak → burst-limited; tens of GB/day typical, TB/day if streamed.-- Oscilloscope: 1 Gbps peak → burst-limited; single-GB to tens of GB/day typical, ~1 TB per workcell on a heavy probing day.+- Oscilloscope: 1 Gbps peak → burst-limited; single-GB to tens of GB on a light day, up to ~1 TB/day. - FLIR: ~0.5–1.5 MB/frame → MB/day for single shots, up to ~40 GB/day continuous. - Microphone: 176.4 kB/s → ~15 GB/day continuous, per mic. - Power sensor: ~15–50 GB/day continuous, per channel.@@ -42,10 +44,15 @@ early 2027. "Active" = 8 h/day of workcell operation. - BMI270: ~140 MB/day at 100 Hz to ~2.2 GB/day at 1.6 kHz (binary); ~10× as JSONL. - BME690: ~1 MB/day at 1 Hz binary; ~90 MB/day at 10 Hz JSONL. -**Node totals:** ~5 TB/day typical at today's 20 workcells, and more on heavy-probing days. ~240 TB/day at the 1,000 workcells projected by 2029. The potential-28 Gbps uplink is ~302 TB/day saturated, which covers about 1,000 workcells; one-sustained 5 Gbps spectrum capture uses about a fifth of it.+**Node totals:** ~24 TB/day at today's 20 workcells (~8.8 PB/year). ~120 TB/day at+the 100 workcells expected by the end of 2027 (~44 PB/year). ~1.2 PB/day at the+1,000 workcells projected by 2029. The potential 28 Gbps uplink is ~302 TB/day+saturated, which covers the raw capture of about 250 workcells; one sustained 5 Gbps+spectrum capture uses about a fifth of it.++Raw capture lands in local caching at the Cloud Lab, and derived metadata is what+gets published. We expect 50:1 to 100:1 compression from raw data to derived+metadata; this has yet to be measured.  ## Publication 
package.json+1−1
@@ -2,7 +2,7 @@   "slug": "pcl-data-profile",   "title": "Adom PCL Node Data Profile",   "type": "skill",-  "version": "0.1.4",+  "version": "0.1.5",   "description": "The physical shape of the data produced by the Adom Industries NSF PCL Test Bed node: every instrument that generates data, the file types each writes, and the rate it writes them at.",   "license": "CC-BY-4.0",   "tags": [
page.json+1−1
@@ -2,7 +2,7 @@   "slug": "pcl-data-profile",   "title": "Adom PCL Node Data Profile",   "type": "skill",-  "version": "0.1.4",+  "version": "0.1.5",   "description": "The physical shape of the data produced by the Adom Industries NSF PCL Test Bed node: every instrument that generates data, the file types each writes, and the rate it writes them at.",   "tags": [     "nsf-pcl",