# octolux-app-debug — debug the Octolux app server

> **Scope.** This skill covers the **Octolux Node.js app server** in `~/ntx-embedded/apps/octolux/` — the thing that serves the Dashboard, Projects, Deploy, VNC, Screencast, and Remote sections on port 8830. Device-side issues (HMI won't boot, Chromium kiosk crashes, Weston compositor, audio, GPIO) belong in the `octolux-platform` skill. Deploy / push-to-device flow belongs in `octolux-deploy`.

## When to use this skill

- User reports the Octolux Dashboard webview is blank, shows a proxy error, or won't load.
- `octolux health` returns anything other than `OK: Octolux server is running`.
- Nothing is bound to port 8830 (`ss -ltn | awk '$4 ~ /:8830/'` is empty).
- User says "the dashboard isn't loading," "the octolux server keeps dying," "port 8830 isn't listening," "ECONNREFUSED on the dashboard."

## Diagnostic flow — start here, every time

**The Hydrogen dashboard tab's `ECONNREFUSED 0.0.0.0:8830` is *always* upstream of the proxy.** It never means the proxy itself is broken; it means the server on the other side of the proxy is not listening. Diagnose upstream:

```bash
# 1. Is the server process alive?
octolux health

# 2. If not, did it crash on startup? Read the log.
cat /tmp/octolux-server.log    # or wherever the server's stdout was redirected

# 3. Is anything on the port?
ss -ltn | awk '$4 ~ /:8830/'

# 4. Did npm install actually run?
ls ~/ntx-embedded/apps/octolux/node_modules/ws 2>/dev/null \
  && echo "ws present" || echo "ws MISSING — run npm install"
```

The log is the source of truth. `octolux health` only tells you *that* the server is dead, not *why*. Skipping straight to "restart it" without reading the log is the #1 mistake — you'll restart it into the same crash and be back where you started.

## Symptoms

Each symptom below has a recognition pattern, the actual root cause, and the fix.

### Symptom 1 — `ECONNREFUSED 0.0.0.0:8830` on the dashboard (and `node_modules/` is missing)

The fresh-container default. Documented in detail because most first-time users hit it.

**Recognition**

| Layer | What you see |
|---|---|
| Webview | Dashboard tab shows only `connect ECONNREFUSED 0.0.0.0:8830` |
| CLI | `octolux health` → `ERROR: Octolux server is not running` |
| Process | Nothing on port 8830 |
| Server log | `Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'ws' imported from .../server.js` |

**Root cause.** `~/ntx-embedded/apps/octolux/node_modules/` was never populated — `ws` and other runtime deps are missing because the container doesn't pre-run `npm install` inside `apps/octolux/`, and there is no top-level `package.json` at `~/ntx-embedded/` that would cover it.

**Fix.**

```bash
cd ~/ntx-embedded/apps/octolux
npm install --no-audit --no-fund

nohup node ~/ntx-embedded/apps/octolux/server.js > /tmp/octolux-server.log 2>&1 &

octolux health
#   → OK: Octolux server is running (octolux v1.0.0)
```

Then refresh the Dashboard tab:

```bash
adom-cli hydrogen webview open-or-refresh --name "Octolux" \
  --url "$(echo $VSCODE_PROXY_URI | sed 's/{{port}}/8830/')dashboard/" \
  --panel-id "$(adom-cli hydrogen workspace tabs 2>/dev/null \
      | python3 -c 'import sys,json; print(json.load(sys.stdin)["tabs"][0]["panelId"])' 2>/dev/null)"
```

### Symptom 2 — `EADDRINUSE 0.0.0.0:8830` (or server log says the port is taken)

**Recognition.** `octolux health` says the server isn't running, but `ss -ltn` shows something IS on port 8830. Starting a new server.js gets `Error: listen EADDRINUSE: address already in use 0.0.0.0:8830`.

**Root cause.** A stale `node server.js` from a previous shell, or a different process that grabbed 8830.

**Fix.** Find the specific PID and kill only that one — **never** `pkill node` (see *Do not do these*):

```bash
pgrep -af 'node.*octolux/server.js'
# → 12345 node /home/adom/ntx-embedded/apps/octolux/server.js
kill 12345
# verify the port is free, then start fresh:
nohup node ~/ntx-embedded/apps/octolux/server.js > /tmp/octolux-server.log 2>&1 &
```

If `ss -ltn` shows 8830 held by a non-`node` process, that's a different problem — figure out what it is before killing it.

### Symptom 3 — `octolux health` is OK but the dashboard is still blank

**Recognition.** Server says it's running. `ss` confirms it's listening on 8830. But the webview tab is still empty or stuck.

**Root cause.** Almost never the server. Two likely culprits:
- The Hydrogen webview is caching the old `ECONNREFUSED` page and hasn't reloaded.
- The browser-side JS in the dashboard page is failing (a CSP / cross-origin / proxy-rewrite issue).

**Fix.** Refresh the tab via `adom-cli hydrogen webview open-or-refresh` (it does a hard reload). If still blank after that, capture the panel for inspection:

```bash
adom-cli hydrogen screenshot panel --name "Octolux" --reason "dashboard blank despite healthy server"
```

Read the screenshot for browser-side errors (often visible as a small text overlay in the corner) before assuming the server is bad.

### Symptom 4 — server starts, then crashes a few seconds later

**Recognition.** `octolux health` is briefly OK, then immediately starts returning errors again. Or `nohup` shows the process exited shortly after launch.

**Root cause.** The server got past dependency loading and crashed inside its own init code — typically a missing env var, a malformed config file, or an uncaught exception inside a route handler that fires on startup.

**Fix.** Tail the log live to catch the actual error:

```bash
tail -f /tmp/octolux-server.log
# in another shell:
nohup node ~/ntx-embedded/apps/octolux/server.js > /tmp/octolux-server.log 2>&1 &
```

Then fix whatever the log surfaces — there is no canonical fix here, it depends on the specific exception.

### Symptom 5 — section buttons (VNC / Deploy / Stream / Camera / Projects) do nothing on first click

A runtime feature bug, not a startup failure. The server is healthy but the dashboard's per-device toolbar buttons appear dead on first interaction. Documented 2026-05-29.

**Recognition.** `octolux health` is OK; the dashboard webview loads and renders the device cards. But clicking a section button on a device card — VNC, Deploy, Stream, Camera — or the top-bar Projects button does nothing the first time. The button looks like it depresses, then nothing visible happens. Curling the underlying endpoint directly returns success:

```bash
PROXY=$(echo $VSCODE_PROXY_URI | sed 's/{{port}}/8830/')
curl -s -X POST "${PROXY}dashboard/api/open-app" \
  -H "Content-Type: application/json" -d '{"app":"vnc"}'
# → {"ok":true,"action":"created","tab":"Octolux VNC"}
```

The server *does* create the tab. The user just can't see it.

**Root cause.** In `~/ntx-embedded/apps/octolux/server.js`, the `/dashboard/api/open-app` handler creates the section tab via `adom-cli hydrogen webview open-or-refresh` but only foregrounds it (via `active-tab`) when the tab already existed. On a *first* click, the new tab is created in the same Hydrogen pane as the dashboard (the server correctly resolves the dashboard's pane via the `panelState.url` heuristic) but is placed *behind* the active dashboard tab. The dashboard stays foreground, so the user concludes the click did nothing.

The original code (look around line 723 of `server.js`):

```javascript
} else {
  const args = ['hydrogen', 'webview', 'open-or-refresh', '--name', targetName, '--url', appUrl];
  if (panelId) args.push('--panel-id', panelId);
  await cli(...args);
  if (tabExists) await cli('hydrogen', 'workspace', 'active-tab', '--name', targetName);  // ← bug
  action = tabExists ? 'navigated' : 'created';
}
```

**Fix.** Always call `active-tab` after `open-or-refresh`, regardless of `tabExists`:

```javascript
} else {
  const args = ['hydrogen', 'webview', 'open-or-refresh', '--name', targetName, '--url', appUrl];
  if (panelId) args.push('--panel-id', panelId);
  await cli(...args);
  // Always foreground the tab — a newly created tab is placed behind whichever tab
  // is already active in the same pane (i.e. the Dashboard), so the user sees no
  // change without this explicit activation.
  await cli('hydrogen', 'workspace', 'active-tab', '--name', targetName);
  action = tabExists ? 'navigated' : 'created';
}
```

Then restart the server with the canonical "kill specific PID, never `pkill`" pattern:

```bash
OCTOLUX_PID=$(pgrep -af 'node.*octolux/server\.js' | awk 'NR==1{print $1}')
[ -n "$OCTOLUX_PID" ] && kill $OCTOLUX_PID
sleep 1
nohup node ~/ntx-embedded/apps/octolux/server.js > /tmp/octolux-server.log 2>&1 &
octolux health
```

**Verification.** After the restart, click any section button and confirm `isActive`:

```bash
adom-cli hydrogen workspace find-tab "Octolux VNC"
# → "isActive": true
```

**Why this was easy to miss.** Once you've clicked a section button once in a session, the tab persists. Future clicks hit the `tabExists` branch and *do* call `active-tab`, so the button "works" — only the very first click is dead. If your dev workflow always opens a section before testing, you never see the bug. This is the kind of regression that wants a smoke test that runs against a *fresh* container state where no section tabs exist yet.

**Scope of the bug.** Affects **all six** sections that route through `/dashboard/api/open-app`: VNC, Deploy, Stream (Screencast), Camera, plus the top-bar Projects button. It was reported as a VNC issue, but the fix applies to the whole class.

### Symptom 6 — section buttons do nothing on *every* click because the Hydrogen browser session is disconnected

Same surface symptom as Symptom 5, completely different root cause. If Symptom 5's fix is in place and clicking a section button *still* does nothing — including subsequent clicks, not just the first one — this is the case. Documented 2026-05-29.

**Recognition.** All section buttons silently fail every time, not just the first click. `octolux health` is OK. The server logs `[open-app] failed: …` lines and the response body — when read against `localhost:8830` directly — contains:

```text
adom-cli hydrogen webview open-or-refresh ... failed:
  {"error": "add-tab returned 409: ..."}
  {"error": "No browser session is connected.
            The editor tab may need to be refreshed.",
   "hint":  "Workspace mutations require a live browser session
             to take effect. Try refreshing the Hydrogen editor tab."}
```

Diagnostic command:

```bash
PROXY=$(echo $VSCODE_PROXY_URI | sed 's/{{port}}/8830/')
curl -s -X POST "${PROXY}dashboard/api/open-app" \
  -H "Content-Type: application/json" -d '{"app":"vnc"}'
# → {"ok":false, "error":"...409 No browser session...", "hint":"Refresh..."}
```

**Root cause (the environmental one).** Hydrogen workspace mutations (`webview open-or-refresh`, `workspace add-tab`, `workspace active-tab`) require a **live browser-side connection** to Hydrogen — i.e., your editor browser tab open and connected to its workspace WebSocket. When that browser session drops (tab closed, network blip, sleep / wake), every workspace mutation from anywhere in the container fails with HTTP 409 from `hydrogen.adom.inc`.

**Root cause (the server bug that hid it).** The original `cli()` wrapper in `~/ntx-embedded/apps/octolux/server.js` `/dashboard/api/open-app` handler resolved with `{err, stdout}` but callers only destructured `stdout` — so subprocess failures were silently dropped and the handler returned `{"ok":true,"action":"created"}` even when nothing had actually been created. The 409 from Hydrogen was completely invisible to the dashboard until you patched the wrapper.

**Fix part 1 — make `cli()` actually surface failures** (the server change):

```javascript
// Before — silently drops errors:
const cli = (...args) => new Promise(r => {
  execFile('adom-cli', args, { timeout: 8000 }, (err, stdout) => r({ err, stdout }));
});

// After — fails loudly when the CLI exited non-zero OR when stdout JSON
// contains an `error` field (Hydrogen 409 surfaces this way: exit-0,
// JSON body {"error": "..."}). Either case rejects so the handler can
// return a proper error to the dashboard.
const cli = (...args) => new Promise((resolve, reject) => {
  execFile('adom-cli', args, { timeout: 8000 }, (err, stdout, stderr) => {
    let parsed = null;
    try { parsed = JSON.parse(stdout || ''); } catch { /* not JSON */ }
    if (err) {
      const msg = (parsed && parsed.error) || (stderr && String(stderr).trim()) || err.message;
      return reject(new Error(`adom-cli ${args.join(' ')} failed: ${msg}`));
    }
    if (parsed && parsed.error) {
      return reject(new Error(`adom-cli ${args.join(' ')} failed: ${parsed.error}`));
    }
    resolve({ stdout, parsed });
  });
});
```

Wrap the body of the handler in `try { ... } catch (e) { return json(res, 200, { ok: false, error: e.message, hint: ... }); }`. Detect the "No browser session is connected" substring and surface a user-facing hint pointing at the editor-tab refresh.

**Fix part 2 — return 200 with `ok:false`, NOT 5xx** (the proxy gotcha):

The Adom container's HTTPS proxy at `*.adom.cloud` strips response bodies on 5xx (Cloudflare-style stock error page). If the server returns `HTTP 502 { ... }` the dashboard only sees `error code: 502` and the hint never reaches the user. Always return HTTP 200 with `{ ok: false, error: ..., hint: ... }` for application-level failures. The dashboard JS must key off `body.ok`, not `r.ok`:

```javascript
const r = await fetch(base + '/api/open-app', { ... });
const body = await r.json().catch(() => ({}));
if (!r.ok || body.ok === false) {
  showToast(`Couldn't open ${action.toUpperCase()}`, body.error, body.hint);
}
```

**Fix part 3 — show the toast on the dashboard** (the visible part):

Add a small toast renderer to `ui/dashboard/index.html` so the user sees the title + error + hint instead of a button that looks dead. The hint is the key payload — it tells them to refresh the editor tab.

**The user-side recovery action.** Once these patches are in place, the user-side fix is always the same: refresh the Hydrogen editor browser tab (right-click → reload, or `Cmd/Ctrl-R`). That re-establishes the workspace session, and the next click works.

**Why this kept reproducing in our session.** Symptom 5's fix to always-call-active-tab was correct, but Hydrogen had silently disconnected (e.g., the editor tab paused in the background or lost its WebSocket). With the old `cli()` wrapper, every `add-tab` / `open-or-refresh` after that point looked like it succeeded — including on the very first click and every subsequent click — so the surface symptom "button looks dead" persisted even though the *original* root cause was fixed. The lesson: **don't ignore subprocess errors**, and **don't rely on 5xx semantics through a body-stripping proxy**.

**Verification.** After applying parts 1+2+3:

```bash
PROXY=$(echo $VSCODE_PROXY_URI | sed 's/{{port}}/8830/')
curl -s -X POST "${PROXY}dashboard/api/open-app" \
  -H "Content-Type: application/json" -d '{"app":"vnc"}'
```

When Hydrogen is disconnected, the response body must contain `"ok":false`, a real `error` string, and `"hint":"Refresh the Hydrogen editor browser tab ..."`. Status code stays 200 so the proxy passes the body through.

## Other ERR_MODULE_NOT_FOUND variants

`ERR_MODULE_NOT_FOUND` for any package name, not just `ws`, has the same root cause as Symptom 1 (missing `node_modules/`). The fix is identical: `npm install` in `~/ntx-embedded/apps/octolux/`. If npm install completes successfully but a *different* module is still missing on the next start, the app's `package.json` is incomplete or out of sync with what `server.js` actually imports — file it as a bug against the app rather than working around it locally.

## Do NOT do these

- **Do not `pkill node`** — VS Code Server runs as Node in this container; killing all Node processes kills the editor and effectively ends the container session (only an Adom admin can restart it). Always find the specific Octolux server PID with `pgrep -af 'node.*octolux/server.js'` and kill only that PID.
- **Do not bounce the container's HTTPS proxy** in response to `ECONNREFUSED 0.0.0.0:8830` — the proxy is working; it's accurately reporting an upstream failure. The fix is on the Octolux server side.
- **Do not assume Octolux deploy or device-side issues share root causes** with app-server issues — they have separate skills (`octolux-deploy`, `octolux-platform`).
- **Do not "fix" the dashboard by restarting the Hydrogen tab repeatedly** — if `ECONNREFUSED` is reproducing, the tab is doing its job. Diagnose the server first.

## Preventing recurrence

Every new container hits Symptom 1 on first dashboard open. Options for cutting that recurrence (ordered cheapest → most invasive):

1. **Document it in `~/.claude/CLAUDE.md` for this project.** Add a line under "Quick Start" or a new "First-run setup" section saying explicitly: *"Run `cd ~/ntx-embedded/apps/octolux && npm install` before the first `octolux health` on a new container."* Cheap, no infrastructure change, surfaces to every Claude thread that loads the project context.
2. **Auto-install on `octolux health` if `node_modules/` is missing.** Modify the `octolux` CLI's `health` command so that, before reporting "server not running," it checks for `node_modules/ws` and if absent prints `HINT: deps are not installed. Run \`cd ~/ntx-embedded/apps/octolux && npm install\` first.` — converts the current bare error into a self-diagnosing one. The CLI already prints a `Hint:` line for the start command; this is the same pattern.
3. **Add the npm install to the container bootstrap.** The gallia installer (`~/gallia/install.mjs`) runs on every fresh container — adding a step that `npm install`s any `~/ntx-embedded/apps/*` that has a `package.json` would eliminate the symptom entirely. More invasive (changes shared infrastructure), but the right long-term fix.
4. **Have `server.js` print a useful error before crashing.** Wrap the top-level imports in a try/catch that detects `ERR_MODULE_NOT_FOUND` and prints `MISSING DEPS — run npm install in this directory` before re-throwing. Less helpful than option 2 because users still have to read the log.

Option 1 is what we did today (via this skill itself, which now ships in the wiki). Option 2 is the right next step if the issue keeps biting people.

## Why this happens (architecture)

The Adom Docker container ships the `~/ntx-embedded/` repo cloned in place but does **not** pre-run `npm install` inside `apps/octolux/`. There is no top-level `package.json` at `~/ntx-embedded/` that could cover it — the Octolux app's `package.json` is local to its own directory.

When the user opens the Octolux Dashboard for the first time via `adom-cli hydrogen webview …`, Hydrogen opens a tab against the in-container HTTPS proxy at `…/proxy/8830/dashboard/`. The proxy hits `0.0.0.0:8830`, finds nothing listening, and renders the error page with body `connect ECONNREFUSED 0.0.0.0:8830`.

If the user follows the `octolux health` hint and runs `node server.js`, Node's ESM resolver fails on the first `import 'ws'` and exits before any port is bound. A `nohup`'d background start swallows the crash output — the user sees the same dashboard error and concludes "the start command didn't work," when in fact the server briefly ran and died.

The fix is therefore always **(a) read the server log to confirm what actually crashed it, then (b) install the missing piece, then (c) start the server again** — never just "try the start command harder."

## File map

| Path | What |
|---|---|
| `~/ntx-embedded/apps/octolux/server.js` | The Node.js app server entry point |
| `~/ntx-embedded/apps/octolux/package.json` | The app's Node dep manifest |
| `~/ntx-embedded/apps/octolux/node_modules/` | Created by `npm install` — must exist for server to start |
| `/tmp/octolux-server.log` | Recommended stdout/stderr redirect for the detached server |
| `/usr/local/bin/octolux` | CLI for `octolux health`, `octolux projects list`, `octolux deploy …`, etc. |
| `/usr/local/bin/adom-cli` | Hydrogen webview / screenshot driver used to open the Dashboard tab |

## Related skills

- `octolux-platform` — device-side issues (Weston, Chromium kiosk, boot sequence, audio, GPIO). Use when the HMI device itself misbehaves, not when the app server misbehaves.
- `octolux-deploy` — push HTML to a device, manage orientation, the deploy API. Use after the server is up and you're trying to ship an interface.
- `octolux-hmi-design` — design the HTML/CSS for an HMI surface.
- `adom-workspace-control` — open / refresh the Octolux Dashboard panel from any agent.