/outline cannot process a large board STEP: hard 170s worker timeout, fails at any memory cap

by adom

The Adom STEP engine: fully manage STEP files. Convert to color-preserving GLB, render thumbnails and vector outlines, laser-etch MPNs, create and edit STEP geometry, extract features, and run molecule mode. One name for the thin Rust CLI and the shared OCCT service. Formerly 'step2glb' + 'service-step2glb'.

Overview Files Skills Discover Packages Releases Dependencies Issues16 Breadcrumbs Pull Requests Activity

Open bug report

Ray · 1mo ago

POST /outline cannot process a large board STEP. The OCCT-HLR worker is run with a hard 170-second subprocess timeout, and a 31.5 MB board exhausts it every time, on a container with 16 GB and four idle workers.

This is a compute-time limit, not a memory limit: the job fails at any memory cap. Raising the container's memory would not change it.

Reproducer

Input: bq25792-charger.step (31.5 MB), the STEP published on adom/bq25792-charger.

curl -X POST "$SERVICE/outline?upAxis=z" -H 'X-Client: …' -H 'X-Job-Name: …' \
  --data-binary @bq25792-charger.step

Every job reports status: failed after ~176 s wall:

outline worker exited 1 | …
subprocess.TimeoutExpired: Command '[…/occt_outline.py, …/in.step, …, 'chip', 'z', '--rot', '180']'
  timed out after 170 seconds

Job ids, all against occt-fbnsqrdkehdj on 1.3.4 dcde0b9a: 873d1a204a6c4eba (single), then 0fea99c3ae14d00c, e40a956704f9268f, 7227d5b0093bde55, e761fba36446cdea (four concurrent).

The job's own hint is good and correctly points away from an input fault:

resource fault (memory/time), not an input fault: re-submit when the service is quieter (GET /health shows workers), or split the model

but re-submitting when quieter does not help, because the limit is per-job wall time, not contention. A single job on an otherwise idle container fails identically.

Where it is

api/occt/outline_job.py:41

r = subprocess.run(cmd, capture_output=True, text=True, timeout=170)

api/occt/name_emboss_job.py:55 carries the same constant, so /name-emboss has the same ceiling.

Measured cost

From a memory-instrumented run on the 16 GB container (cgroup memory.current, sampled 4×/s):


peak, one job	2.88 GB
peak, four concurrent jobs	10.12 GB of the 16 GB cap
swap growth across the run	+6.9 MB (negligible)
wall time per job	~176 s, all four hitting the 170 s cap

So the work is memory-feasible and simply does not finish. Four ran side by side with 5.9 GB to spare.

Please do not just raise the constant

The right question is where the time goes. OCCT HLR on a model with this many faces is the likely sink, and /outline renders 16 styles, so a per-style cost is multiplied. Worth measuring before changing the number: per-style timing, whether styles share one HLR projection or recompute it, and whether a face-count threshold should decimate or refuse with a clear message rather than burn 170 s and fail.

Relationship to #9 — different fault, do not merge them

#9 (John) is not this. His three failing files are ~0.3 MB and report "unknown error" with duration_seconds: null, i.e. they die immediately with empty stderr — the signal/segfault path described in the reply on that issue. This one is a 31.5 MB file that runs the full 170 s and fails with a named TimeoutExpired. A 0.3 MB part cannot exhaust a 170 s HLR budget; a 0.2 MB part outlines in a couple of seconds on the same service. Same endpoint, two distinct mechanisms.

Found while running the standby-container endpoint parity diff; both containers fail this identically, so it is not a standby regression.

2 Replies

Ray · 1mo ago

Measured where the 170 s goes. It is not the sixteen styles, and it is not an algorithmic hot spot we can tune away — it is single-threaded OCCT HLR on a host about six times slower per core than a dev box.

The style loop is innocent. hlr() is called once and all eleven styles reuse its vis/hid polylines; nothing re-projects per style. Timing the whole eleven-style SVG loop on the board's 82,652 polylines: 0.9 s. Rewriting it to join a list instead of body += gains 1.1x, and precomputing each point's formatted string once and reusing it across styles gains 3.7x — both real, both irrelevant next to a 170 s budget.

Where it actually goes, whole pipeline on the 31.5 MB board, no timeout:

stage	time
read file	1.6 s
transfer roots	13.5 s
to_view	1.2 s
hlr	46.1 s
11-style SVG loop	0.9 s
total	~63 s, peak RSS 2.50 GB

That board is 711 solids, 24,659 faces, 133,422 edges, producing 14,958 visible and 67,694 hidden polylines.

Sixty-three seconds is comfortably inside the 170 s cap — on a dev box. The job still times out on the service every time, so the difference is the machine.

The host penalty, measured rather than assumed. Same model, same code, same harness, run on the service container under its own venv and on a dev container:

	HLR	total
dev container (Core Ultra 9 285H)	0.4 s	0.7 s
service container (own venv)	2.4 s	4.2 s

6.0x slower, and HLR measures parallelism 1.00x on both — it is strictly single-threaded, so the container's CPU quota (2.0 on production, 4.0 on the twin) is not what binds a single job; single-core speed is. Scale the board's ~63 s by that factor and it needs roughly 350 s, which is why it dies at 170 s with room to spare on memory.

So raising the constant is the wrong move, for three reasons: the number would have to go past 350 s for this one board and would not generalise; HLR is single-threaded, so a job would hold a worker slot for six minutes while three sit idle; and the cost scales with edge count, so the next larger board moves the goalposts again.

Worth doing instead, roughly in order of value:

Refuse early and honestly. Count edges after transfer and, above a threshold, fail immediately with the count and the limit. Today the job burns 170 s of a worker slot and then emits a hint saying "re-submit when the service is quieter" — which is misleading, because a single job on a completely idle container fails identically. That hint should not survive this bug.
Cut the input instead of the clock. An outline is a silhouette; it does not need all 711 solids. Culling to the board body and the top-level components, or decimating below a size threshold, attacks the 133,422 edges directly, which is where the time is.
Keep the two SVG micro-optimisations for whenever that file is touched; they are free, but they are not the fix.

One incidental: the two OCP builds disagree slightly on tessellation — the same molecule yields 1505/2301 polylines on the service and 1458/2308 on the dev box. Not a problem here, but worth knowing before anyone byte-compares outline output across machines.

/name-emboss carries the same 170 s constant at api/occt/name_emboss_job.py:55 and will have the same ceiling.

Ray · 1mo ago

First half shipped: 1.3.6 (8c57cf13), live. The job is now refused up front instead of consuming the whole budget and timing out. Leaving this open for the second half.

On production, the same 31.5 MB board:

model too large for this worker: 133422 edges against a 50770 limit on this machine
(estimated 291.7s of HLR, 95.1s of budget left, machine 6.32x slower than reference).
OCCT HLR is single-threaded and superlinear in edge count, so a retry will not pass
and a quieter service makes no difference. Reduce or split the model.

	before	after
wall time	176 s, timeout	81 s
worker slot held	the full budget	released as soon as the size is known
what the caller is told	"re-submit when the service is quieter"	the edge count, the limit, the estimate and why waiting cannot help

A normal part on the same service is unaffected: 11 styles, 120,845 bytes.

The limit is calibrated, not hard-coded. Cost is modelled as hlr_s = 3.426e-6 × edges^1.391 — it has to be superlinear, because 4,550 edges run at 10,842 edges/s while 133,422 run at 2,892, so no single rate is right at both ends. The machine factor comes from one small deterministic HLR timed against what that same shape costs on the reference machine. Production measured itself at 6.32x, against the 6.0x I had measured independently on the twin, which is a satisfying agreement. On a dev box the factor lands near 1.0 and the very same board is correctly not refused, because it finishes there in 51 s. That is the point of calibrating rather than picking a number.

Why 81 s and not instant: you cannot count a model's edges without reading and transferring it, and that alone is ~75 s for 31.5 MB on this hardware. The refusal happens the moment the size is known. Worth stating plainly rather than claiming an immediate rejection.

One bug found while proving it, which is why there is a 1.3.6. In 1.3.5 the refusal fired correctly but the job came back with the generic hint. The queue keeps only the last 400 characters of a worker's stderr, and 1.3.5 wrote the machine-readable marker line first and the prose second, so truncation ate the marker and the specific hint never matched. The tell was that the production error text began mid-sentence. It is now one 334-character line ending with the marker, and the tests pin all three properties: it fits the window, the marker survives a tail truncation, and the marker is last.

Also deleted: the hint that said "re-submit when the service is quieter (GET /health shows workers)". That advice was false — an idle container fails identically — and it is the kind of thing that costs someone an afternoon. SIGKILL and timeout now carry separate accurate hints.

/name-emboss shares the 170 s constant and got the same guard.

Still open — the second half. This stops the service wasting three minutes to tell you no. It does not let the board through. Doing that means culling or decimating the input, since an outline is a silhouette and does not need all 711 solids, and that is a product decision about what a board outline should contain rather than a purely technical one. Going to Ray with the numbers.

PR: https://github.com/adom-inc/service-occt/pull/10