# web-serial-hmi — NTX HMI ↔ USB CDC firmware pattern

> **Scope.** Building a Chromium-kiosk HMI on an **NTX Octolux device** that auto-connects to a **USB-attached RP2040 or RP2350** target over CDC-ACM, with **no Web Serial picker** ever showing. End-to-end working pattern, not a per-customer recipe — instantiate it for any new project under `~/ntx-embedded/customers/<id>/`.
>
> Out of scope: Android USB Host (different stack); Web Bluetooth / WebHID; non-RP-series USB devices (mostly the same pattern, just substitute the VID); CAN-FD bridges; the SWD/flash side of the firmware lifecycle — that's covered by `rp2350-flash` and the `target/rp2040.cfg` openocd patterns and the local `firmware/` build is just plain `cmake + make`.

## When to use this skill

- Starting a new customer HMI project that needs to control hardware over a USB-attached MCU.
- Porting an Android-tablet HMI app to an Octolux Linux/Chromium device and you want the same "just plug it in" feel.
- The Web Serial picker is showing — *"No compatible devices found"*, *"No port selected by user"*, or just appearing at all — and the project shouldn't have a picker.
- `requestPort()` works once but the operator has to tap-and-pick every session and that's wrong for a kiosk.
- The HMI page can't see what's happening because Chromium permission dialogs don't render in VNC.

## Architecture

```
┌─────────────────────────────────┐      USB-C       ┌──────────────────────┐
│ NTX Octolux device (10.0.10.x)  │ ◄──────────────► │ RP2040 / RP2350 board│
│ ─────────────────────────────── │ CDC-ACM, 115200  │ ───────────────────  │
│ Chromium 109 kiosk @ localhost  │   8N1 line ASCII │ Pico SDK USB CDC     │
│  ├─ Web Serial API              │                  │  ├─ getchar polling  │
│  ├─ SerialAllowAllPortsForUrls  │                  │  ├─ ASCII line parser│
│  └─ getPorts() filtered by VID  │                  │  └─ GPIO drive       │
│ Apache at /                     │                  │                      │
│ /dev/ttyACM0 (root:root, 0666)  │                  │                      │
└─────────────────────────────────┘                  └──────────────────────┘
        ▲                                                       ▲
        │  SSH from container (deploy + udev + policy)          │  SWD via PiProbe (flash)
        │                                                       │
   Adom Docker container                                  see rp2350-flash
   ~/ntx-embedded/customers/<id>/{hmi,firmware}
```

Two physical links from the workcell to the target board:

- **USB-C** between Octolux and target — runtime command path (this skill).
- **SWD** between the workcell's PiProbe and the target — flash path (`rp2350-flash`).

The container is the build host for both: HMI HTML edits go SCP-ed to Octolux; firmware ELF is built locally and flashed via the PiProbe.

## Required components

A working NTX HMI+firmware pair has **five** moving parts. Skip any one and connect won't happen. Pieces 3 and 4 are one-time setup *per Octolux device*; pieces 1, 2, 5 are per-project.

| # | Layer | Where it lives | Per-… |
|---|---|---|---|
| 1 | **Firmware** — USB CDC stdio, command parser, GPIO drive | `customers/<id>/firmware/src/main.c` (in container) | project |
| 2 | **HMI HTML** — connect logic + UI | `customers/<id>/hmi/index.html` (in container, deployed to device) | project |
| 3 | **Chromium policy** — `SerialAllowAllPortsForUrls` | `/etc/chromium/policies/managed/octolux-serial.json` (on device) | device |
| 4 | **udev rule** — `MODE="0666"` for VID 0x2e8a | `/etc/udev/rules.d/99-rp-cdc-acm.rules` (on device) | device |
| 5 | **Deploy** — SCP HMI + chromium restart + screencast bounce | shell pattern run from container | project |

## Quick start — new project, end to end

Run from the Adom container. Substitute `<id>` (e.g. `acme-control`), `<device-ip>` (e.g. `10.0.10.98`), and your domain commands.

### 1. Scaffold

```bash
ID=acme-control
DEVICE_IP=10.0.10.98
DEVICE_MODEL='Octolux 15.6"'
VIEW_W=1920 VIEW_H=1080

mkdir -p ~/ntx-embedded/customers/$ID/{hmi,firmware/src}

cat > ~/ntx-embedded/customers/$ID/project.json <<EOF
{
  "id": "$ID",
  "name": "Acme Control",
  "customer": "Acme",
  "summary": "One-line description of what this HMI does.",
  "accent": "#e87a1e",
  "accentDeep": "#a64f10",
  "deviceModel": "$DEVICE_MODEL",
  "viewport": { "w": $VIEW_W, "h": $VIEW_H },
  "orientation": "landscape",
  "defaultIP": "$DEVICE_IP",
  "tagline": "one-line operator-facing tagline"
}
EOF
```

### 2. Drop in the firmware + HMI templates

Use the templates further down this doc verbatim, customizing only:

- **Firmware:** `FW_VERSION_STRING` (the `<NAME> FW x.y.z` boot banner), the `dispatch()` command list, and your specific GPIO pin numbers / domain logic.
- **HMI:** the visible controls (buttons, sliders, etc.), the brand colors from your `project.json`, the response-line handlers in `handleResponse()`.

Keep the Web Serial connect logic and the visible debug strip **unchanged** — that's the point of this skill.

### 3. Build + flash the firmware (first time)

```bash
export PICO_SDK_PATH=$HOME/pico-sdk
cd ~/ntx-embedded/customers/$ID/firmware
mkdir -p build && cd build && cmake .. && make -j$(nproc)

# Flash via PiProbe over SWD — pick the cfg for your chip:
openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg \
  -c "adapter speed 5000; program ${ID//-/_}.elf verify reset exit"
# (or target/rp2350.cfg for an RP2350 — see rp2350-flash)
```

### 4. One-time device setup (Chromium policy + udev rule)

Skip if this Octolux device has been set up for Web Serial before. Verify by SSH'ing in: if `/etc/chromium/policies/managed/octolux-serial.json` and `/etc/udev/rules.d/99-rp-cdc-acm.rules` both exist, you're done.

```bash
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
set -e
# Chromium policy: SerialAllowAllPortsForUrls turns getPorts() into a
# blanket enumeration for http://localhost; DefaultSerialGuardSetting=2
# also disables requestPort() pickers as a safety net.
mkdir -p /etc/chromium/policies/managed
cat > /etc/chromium/policies/managed/octolux-serial.json <<'POL'
{
  "SerialAllowAllPortsForUrls": ["http://localhost"],
  "DefaultSerialGuardSetting": 2
}
POL

# udev: /dev/ttyACM* is root-only by default on the Yocto distro, and
# the kiosk Chromium can't open it. MODE=0666 fixes that for any
# Raspberry Pi (RP2040/RP2350) USB CDC device.
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'UDEV'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
UDEV
udevadm control --reload-rules

# Re-process any ALREADY-plugged-in device so the new rule applies now.
# (Without this, the perms only apply to *future* plug events.)
BUSPORT=$(lsusb -t 2>/dev/null | awk '/Pico|2e8a/ {sub(/^[ \t]+/, ""); print}' | head -1)
echo "USB device tree row: ${BUSPORT:-(none seen)}"
# Cycle the parent USB device (substitute bus-port from `lsusb -t` if your
# topology differs; 4-1 is the common Octolux internal hub port).
echo 4-1 > /sys/bus/usb/drivers/usb/unbind 2>/dev/null || true
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind 2>/dev/null || true

systemctl restart octolux-chromium
REMOTE

# Per the [[hmi-deploy-reset-vnc]] rule: chromium restart leaves the
# screencast bound to a dead CDP target. Bounce the screencast on the
# device — see the Deploy sequence below for the exact commands.
```

### 5. Deploy the HMI to the device

See the "Standard deploy sequence" section below — it's the same shell pattern you'll run on every HMI edit.

### 6. Verify it auto-connected (CDP eval from container)

```bash
ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP

python3 - <<'PY'
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def evx(expr, await_=False):
    ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
        "params": {"expression": expr, "returnByValue": True,
                   "awaitPromise": await_}}))
    while True:
        r = json.loads(ws.recv())
        if r.get("id") == 1: return r["result"]["result"]["value"]
print("connected:", evx("state.connected"))
print("fwVersion:", evx("state.fwVersion"))
print("ports    :", evx("navigator.serial.getPorts().then(p => p.length)", await_=True))
PY
```

Expected once the firmware is running and the cable is plugged in:

```
connected: True
fwVersion: 1.0.0
ports    : 9          # 8 platform UARTs + your 1 RP-series CDC
```

If `connected: True` and `fwVersion` is populated → you're done; the HMI is talking to the MCU. If not, jump to the failure table.

## Firmware template (RP2040 / RP2350 USB CDC)

`firmware/CMakeLists.txt`:

```cmake
cmake_minimum_required(VERSION 3.13)
set(PICO_BOARD pico CACHE STRING "")            # or "pico2" for RP2350
set(PICO_PLATFORM rp2040 CACHE STRING "")       # or "rp2350"
include($ENV{PICO_SDK_PATH}/external/pico_sdk_import.cmake)
project(<id_underscored> C CXX ASM)
pico_sdk_init()
add_executable(<id_underscored> src/main.c)
target_link_libraries(<id_underscored> pico_stdlib hardware_gpio)
pico_enable_stdio_usb(<id_underscored>  1)      # CDC-ACM on USB-C
pico_enable_stdio_uart(<id_underscored> 0)      # no UART — USB only
pico_add_extra_outputs(<id_underscored>)
```

`firmware/src/main.c`:

```c
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdarg.h>
#include "pico/stdlib.h"
#include "pico/stdio_usb.h"      // stdio_usb_connected()
#include "hardware/gpio.h"

#define FW_VERSION_STRING "<NAME> FW 1.0.0"
#define CMD_BUF_SIZE 96

// Status LEDs (wire your board's status LEDs to these or change as needed).
// LED_USB: solid ON while USB CDC is connected to a host.
// LED_ACT: 50 ms blink on every valid command.
// LED_ERR: solid ON when the last response was ERR; clears on next OK.
// LED_IDLE: solid ON when system is in its safe / inactive state.
#define LED_USB   9
#define LED_ACT  10
#define LED_ERR  16
#define LED_IDLE 18

static char cmd_buf[CMD_BUF_SIZE];
static int  cmd_len = 0;
static bool act_on = false;
static absolute_time_t act_off_at;

static void blink_activity(void) {
    gpio_put(LED_ACT, 1);
    act_on = true;
    act_off_at = make_timeout_time_ms(50);
}
static void say_ok(const char *fmt, ...) {
    va_list a; va_start(a, fmt);
    fputs("OK ", stdout); vprintf(fmt, a); fputs("\r\n", stdout);
    va_end(a);
    gpio_put(LED_ERR, 0);
    blink_activity();
}
static void say_err(const char *m) {
    printf("ERR %s\r\n", m);
    gpio_put(LED_ERR, 1);
    blink_activity();
}
static void to_upper(char *s) { for (; *s; s++) *s = (char)toupper((unsigned char)*s); }

// === Domain commands ====================================================
//
// Replace these with the commands your project actually needs. Keep
// the canonical built-ins (PING, VERSION, STATUS, HELP) — the HMI
// template's handshake assumes they exist.
//
static void dispatch(char *line) {
    char *tok = strtok(line, " \t");
    if (!tok) return;

    if (!strcmp(tok, "PING"))    { say_ok("PONG"); return; }
    if (!strcmp(tok, "VERSION")) { printf("%s\r\n", FW_VERSION_STRING); blink_activity(); return; }
    if (!strcmp(tok, "STATUS"))  { /* print one-line state snapshot */ blink_activity(); return; }
    if (!strcmp(tok, "HELP"))    { /* print "OK …" lines */ blink_activity(); return; }

    // your-domain command parsing here, e.g.:
    //   if (!strcmp(tok, "DO")) { … say_ok("DO %d", arg); return; }

    say_err("unknown command");
}

// === Boilerplate main =================================================

int main(void) {
    stdio_init_all();

    // Init your GPIOs here (LEDs + outputs). Drive outputs LOW first
    // so the system boots in a safe state regardless of host behavior.
    const uint leds[] = { LED_USB, LED_ACT, LED_ERR, LED_IDLE };
    for (size_t i = 0; i < sizeof(leds)/sizeof(leds[0]); i++) {
        gpio_init(leds[i]); gpio_set_dir(leds[i], GPIO_OUT); gpio_put(leds[i], 0);
    }
    gpio_put(LED_IDLE, 1);

    bool was_conn = false, banner_done = false;

    while (true) {
        bool conn = stdio_usb_connected();
        gpio_put(LED_USB, conn ? 1 : 0);
        if (conn && !was_conn) banner_done = false;
        was_conn = conn;

        // Boot banner on every CDC connect — the HMI parses this to
        // populate the FW version pill.
        if (conn && !banner_done) {
            sleep_ms(50);                     // give the host's reader a beat
            printf("%s\r\n", FW_VERSION_STRING);
            puts("TYPE 'HELP' FOR COMMANDS");
            banner_done = true;
        }

        // Drain available input into cmd_buf; dispatch on EOL.
        for (int budget = 0; budget < 32; budget++) {
            int c = getchar_timeout_us(0);
            if (c == PICO_ERROR_TIMEOUT) break;
            if (c == '\n' || c == '\r') {
                if (cmd_len > 0) {
                    cmd_buf[cmd_len] = '\0';
                    to_upper(cmd_buf);
                    dispatch(cmd_buf);
                    cmd_len = 0;
                }
            } else if (cmd_len < CMD_BUF_SIZE - 1) {
                cmd_buf[cmd_len++] = (char)c;
            } else { cmd_len = 0; say_err("command too long"); }
        }

        // Activity LED timeout.
        if (act_on && absolute_time_diff_us(get_absolute_time(), act_off_at) <= 0) {
            gpio_put(LED_ACT, 0); act_on = false;
        }
    }
}
```

### Protocol conventions that work well

- **Newline-terminated ASCII** — accept `\n` and `\r\n`. Case-insensitive (uppercase before dispatch).
- **One line in, exactly one line out** — `OK …`, `ERR …`, or a single-token status line. One-to-one keeps the HMI's send/receive trivial.
- **Standard built-ins always present:**
  - `PING → PONG` (liveness)
  - `VERSION → <NAME> FW x.y.z` (capability detection + version pill)
  - `STATUS → STATE …` (one-line state snapshot)
  - `HELP → multi-line OK …` (operator self-discovery)
- **Boot banner on every CDC connect.** Print `<NAME> FW x.y.z\r\n` first thing after `stdio_usb_connected()` flips true. The HMI matches `/^([\w-]+) FW (\S+)/i` to populate the version pill.
- **LED1 (USB) mirrors `stdio_usb_connected()`** — instant visual confirmation that the host opened the port.
- **LED2 (activity) blinks 50 ms on every valid command** — independent visual signal that the firmware *parsed* a command.
- **Default-safe on boot** — all outputs LOW, critical actuators OFF, idle LED ON. Don't rely on the host to send `ALL OFF` first.

## HMI template

```html
<script>
"use strict";

const state = {
  port: null, writer: null, reader: null,
  connected: false, fwVersion: null,
  autoConnectTimer: null,
};

// ─── Visible debug strip ──────────────────────────────────────────────
// Mirror every UI event + Web Serial step into the page DOM, because
// Chromium permission dialogs (the Web Serial picker, geo prompts,
// etc.) render above the page surface and DON'T appear in the
// screencast / VNC feed. Without this strip, you'd be blind to what's
// happening on the kiosk from your dashboard.
function dbg(line) {
  const strip = document.getElementById('debugStrip');
  if (!strip) return console.log(line);
  const d = document.createElement('div');
  d.textContent = new Date().toISOString().slice(11,23) + '  ' + line;
  strip.appendChild(d);
  while (strip.children.length > 5) strip.removeChild(strip.firstChild);
}

// ─── Line-splitter transformer for the reader stream ──────────────────
function lineSplitter() {
  let buf = '';
  return {
    transform(chunk, ctrl) {
      buf += chunk;
      let i;
      while ((i = buf.search(/\r?\n/)) >= 0) {
        ctrl.enqueue(buf.slice(0, i));
        buf = buf.slice(i + (buf[i] === '\r' ? 2 : 1));
      }
    },
  };
}

async function openExisting(port) {
  await port.open({
    baudRate: 115200, dataBits: 8, stopBits: 1, parity: 'none',
    bufferSize: 256, flowControl: 'none',
  });
  state.port = port;

  const enc = new TextEncoderStream();
  enc.readable.pipeTo(port.writable);
  state.writer = enc.writable.getWriter();

  const dec = new TextDecoderStream();
  port.readable.pipeTo(dec.writable);
  state.reader = dec.readable
    .pipeThrough(new TransformStream(lineSplitter()))
    .getReader();

  state.connected = true;
  readLoop();
  handshake();
}

async function tryAutoConnect() {
  if (state.connected || !('serial' in navigator)) return state.connected;
  const ports = await navigator.serial.getPorts();
  // The kiosk policy returns the FULL set of /dev/tty* devices the
  // kernel sees — typically ~8 platform UARTs (empty getInfo()) plus
  // your real USB CDC. Always filter; never open ports[0] blindly.
  // 0x2e8a is the Raspberry Pi VID (RP2040 + RP2350). Substitute your
  // chip's VID if you use a non-RP target.
  const rp     = ports.filter(p => (p.getInfo() || {}).usbVendorId === 0x2e8a);
  const anyUsb = ports.filter(p => typeof (p.getInfo() || {}).usbVendorId === 'number');
  const candidates = rp.length ? rp : anyUsb;
  dbg(`getPorts() → ${ports.length} total, ${rp.length} RP-series`);
  for (const port of candidates) {
    try {
      await openExisting(port);
      const info = port.getInfo();
      dbg(`auto-connected to VID:${(info.usbVendorId||0).toString(16)} PID:${(info.usbProductId||0).toString(16)}`);
      return true;
    } catch (e) {
      dbg('  open failed on one candidate: ' + e.message);
    }
  }
  return false;
}

window.addEventListener('load', async () => {
  // Install global capture-phase event listeners so we can see every
  // tap on the page, even ones that don't land on a button.
  ['pointerdown','pointerup','touchstart','touchend',
   'mousedown','mouseup','click'].forEach(k => {
    document.addEventListener(k, e => {
      const tgt = e.target ? (e.target.id || e.target.tagName) : '?';
      dbg(`${k}  (${(e.clientX||0)|0},${(e.clientY||0)|0}) → ${tgt}  trusted=${e.isTrusted}`);
    }, /* useCapture */ true);
  });
  dbg('debug listeners installed');

  if (!(await tryAutoConnect())) {
    // Keep polling — if the cable is plugged in *after* the page loads,
    // the next tick picks it up. Cheap (one promise every 2 s).
    state.autoConnectTimer = setInterval(async () => {
      if (await tryAutoConnect() && state.autoConnectTimer) {
        clearInterval(state.autoConnectTimer);
        state.autoConnectTimer = null;
      }
    }, 2000);
  }
});

async function readLoop() {
  while (state.reader) {
    try {
      const { value, done } = await state.reader.read();
      if (done) { state.connected = false; break; }
      if (value) handleResponse(value.trim());
    } catch { state.connected = false; break; }
  }
}

async function handshake() {
  await send('PING');
  await send('VERSION');
  await send('STATUS');
}

async function send(cmd) {
  if (!state.writer) return;
  dbg('→ ' + cmd);
  await state.writer.write(cmd + '\n');
}

function handleResponse(line) {
  dbg('← ' + line);
  const m = line.match(/^([\w-]+) FW (\S+)/i);
  if (m) state.fwVersion = m[2];
  // Your domain handlers here (parse STATUS reply, react to OK/ERR, etc.)
}
</script>
```

The HMI's HTML body and CSS are entirely up to your project — see `octolux-hmi-design` for the brand-language guide (dark mode, accent colors, touch-target sizes, Inter font). The only DOM element this template requires is `<div id="debugStrip"></div>` somewhere visible during development. You can hide it (or remove the global tap listeners) after the project ships.

## One-time Chromium policy — the single biggest enabler

> Without `SerialAllowAllPortsForUrls`, `requestPort()` shows a picker, the picker isn't visible through VNC, and the kiosk is unreliable. With it, `getPorts()` returns every kernel serial device with no user gesture.

```json
{
  "SerialAllowAllPortsForUrls": ["http://localhost"],
  "DefaultSerialGuardSetting": 2
}
```

- **`SerialAllowAllPortsForUrls: ["http://localhost"]`** — pages served from `http://localhost` (which is where the Octolux's Apache serves the HMI) get blanket permission to enumerate and open serial ports. `getPorts()` now returns the full list.
- **`DefaultSerialGuardSetting: 2`** — `2` means *"Do not allow any site to request access"*. Belt-and-suspenders: if your code accidentally calls `requestPort()`, the picker is a no-op instead of presenting UI nobody can see.

Path: `/etc/chromium/policies/managed/octolux-serial.json`. Restart Chromium after install (`systemctl restart octolux-chromium`). After restart, you **must** also bounce the screencast (see `hmi-deploy-reset-vnc`).

Verify the policy is live via CDP eval:

```javascript
navigator.serial.getPorts().then(p => p.length)
// Expected: a positive integer (typically 9 on Octolux 15.6") when an RP
// device is plugged in. Zero means the policy didn't load — re-check
// the file path and the chromium restart.
```

## One-time udev rule

The Octolux Yocto distro ships `/dev/ttyACM0` as `-rw-r--r-- root:root`. The kiosk Chromium can't open it. Fix with a rule that targets the RP-series VID:

```bash
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'EOF'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
EOF
udevadm control --reload-rules
```

For an **already-plugged** device, the rule doesn't retroactively re-permission. Either physically unplug+replug, or cycle the USB driver from a shell:

```bash
echo 4-1 > /sys/bus/usb/drivers/usb/unbind
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind
```

(`4-1` is the typical Octolux internal hub port; substitute from `lsusb -t` if your topology differs.)

Note: the Yocto distro lacks the `usb_id` udev helper, so `udevadm info /dev/ttyACM0` returns *"Unknown device: Inappropriate ioctl"*. **That's fine.** Chromium reads `getInfo().usbVendorId` from the underlying USB descriptor, not from udev's metadata database. You only need the permission rule, not metadata population.

## Standard deploy sequence

Every HMI edit goes through this. Don't skip the screencast bounce — see `hmi-deploy-reset-vnc`.

```bash
ID=acme-control
DEVICE_IP=10.0.10.98

# 1. Push the HTML
scp ~/ntx-embedded/customers/$ID/hmi/index.html \
    root@$DEVICE_IP:/usr/share/apache2/default-site/htdocs/index.html

# 2. Reload the kiosk — restart chromium pulls the new HTML AND
#    reloads any new chromium policy / udev rule deployed alongside.
ssh root@$DEVICE_IP 'systemctl restart octolux-chromium'

# 3. Wait for the kiosk to come back
sleep 10

# 4. Bounce the screencast — chromium restart killed its CDP target
#    binding and the screencast won't auto-reattach. Without this,
#    the dashboard VNC tile freezes on a pre-deploy frame.
#    NOTE: busybox `ps aux` truncates cmdlines so `ps aux | grep
#    screencast` returns NOTHING even when it's running. Walk /proc
#    instead. And NEVER walk /proc/PID/net/tcp to find port owners —
#    those views are namespace-wide and you will mass-kill systemd.
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
for d in /proc/[0-9]*; do
  pid=${d##*/}
  grep -q 'screencast' "$d/cmdline" 2>/dev/null && kill -9 "$pid" 2>/dev/null
done
sleep 2
cd /tmp && nohup node screencast.js --quality 80 --pipeline 4 > /tmp/screencast.log 2>&1 &
REMOTE

# 5. (optional) confirm the dashboard VNC is alive — successive snaps
#    should differ unless the page is genuinely static.
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/a.jpg -w "%{size_download}\n"
sleep 2
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/b.jpg -w "%{size_download}\n"
```

If `/tmp/screencast.js` or `/tmp/node_modules/ws/` ever go missing on the device (they live in tmpfs and can be wiped by a bad kill spree), SCP them back from the container:

```bash
scp ~/ntx-embedded/apps/octolux/ui/screencast/screencast.js \
    root@$DEVICE_IP:/tmp/screencast.js
cd ~/ntx-embedded/apps/octolux/node_modules && tar cz ws | \
  ssh root@$DEVICE_IP 'mkdir -p /tmp/node_modules && cd /tmp/node_modules && tar xz'
```

## Live diagnostics — CDP from the container

The Octolux kiosk exposes Chrome DevTools at `localhost:9222`. Tunnel it back:

```bash
ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP

# install python websocket client once
pip install --user --break-system-packages websocket-client
```

Then any time you want to inspect the live page state:

```python
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def cdp(method, params=None, _id=[0]):
    _id[0] += 1
    ws.send(json.dumps({"id": _id[0], "method": method, "params": params or {}}))
    while True:
        r = json.loads(ws.recv())
        if r.get("id") == _id[0]: return r
def evx(expr, await_=False):
    return cdp("Runtime.evaluate", {"expression": expr, "returnByValue": True, "awaitPromise": await_})["result"]["result"]["value"]

# Useful queries:
evx("state.connected")
evx("state.fwVersion")
evx("navigator.serial.getPorts().then(p => p.length)", await_=True)
evx("Array.from(document.querySelectorAll('#debugStrip > div')).map(d=>d.textContent).join('\\n')")

# Force a reload (and remember: bounce screencast afterwards!):
cdp("Page.reload", {"ignoreCache": True})
```

### Hit-test if a tap doesn't fire the button

```javascript
const b = document.getElementById('btnConnect');
const r = b.getBoundingClientRect();
const hit = document.elementFromPoint(r.left + r.width/2, r.top + r.height/2);
JSON.stringify({ same: hit === b, hit_id: hit?.id, hit_tag: hit?.tagName });
```

If `same` is false, something is covering the button.

## Failure-mode reference

Every row in this table corresponds to a real failure observed during this pattern's development. Use it as the diagnostic flow.

| Symptom | Root cause | Fix |
|---|---|---|
| Picker shows *"No compatible devices found"* | Chromium policy not in place AND/OR `/dev/ttyACM*` not world-readable | Install both the policy and the udev rule; restart chromium; bounce screencast |
| Picker shows the device but every connect ends in *NotFoundError: No port selected by user* | User dismissed the picker, or `requestPort()` was called from a non-user-gesture context (programmatic CDP `.click()` triggers this) | Don't use `requestPort()` at all — use the `getPorts()` auto-connect pattern in the HMI template |
| `getPorts()` returns ports but `port.open()` throws *Failed to execute 'open' on 'SerialPort': Failed to open serial port* | You opened `ports[0]`; it's a platform UART without a backing device | Filter by `getInfo().usbVendorId === 0x2e8a` before opening |
| `getPorts().length === 0` even though `lsusb` shows the device on the kernel side | Policy file missing, malformed, or wrong path; OR Chromium wasn't restarted after the policy install; OR `/dev/ttyACM0` perms are still root-only | Re-check `/etc/chromium/policies/managed/octolux-serial.json` is valid JSON; restart chromium; `ls -la /dev/ttyACM0` should be `0666` |
| Tap on a button does nothing — no visible page reaction | Touch input might not be reaching the page; check via the debug strip (it should log `pointerdown` / `click`). If those don't appear → input layer (Weston/touchscreen) issue. If they do appear but `connect() called` doesn't → the click hit something covering the button — use `elementFromPoint` | Diagnose via debug strip + hit-test |
| The debug strip shows `connect() called → requestPort()...` and hangs — then eventually fails *"No port selected by user"* | The HMI still has `requestPort()` somewhere | Replace with `getPorts()` + VID filter (HMI template) |
| Page reloaded via CDP `Page.reload` — VNC dashboard tile froze on the pre-reload frame | The screencast bound to the old CDP target ID; reload creates a new one; the binding is dead | Bounce screencast (Standard deploy sequence step 4). Applies to ANY Chromium target change — restart, reload, navigate |
| `octolux deploy push` times out | Octolux server's deploy API is flaky on this build | Use the SCP + restart bash sequence directly (it's what the API does internally anyway) |
| `ps aux \| grep screencast` returns nothing but screencast IS running and bound to port 8085 | Busybox `ps aux` truncates cmdlines | Walk `/proc/[0-9]*/cmdline` to find PIDs by argv match |
| Killed a system PID and lost SSH / had to recover the device | You walked `/proc/[0-9]*/net/tcp` to find owners of port 8085 — those views are namespace-wide and match every PID | **NEVER** identify port owners via `/proc/PID/net/tcp` on this distro. Use `/proc/PID/cmdline` matching instead |
| `/tmp/screencast.js` or `/tmp/node_modules/ws/` missing after a bad kill / reboot | They live in tmpfs and the original placement path doesn't auto-restore | SCP from `~/ntx-embedded/apps/octolux/ui/screencast/screencast.js` and tar+SCP `~/ntx-embedded/apps/octolux/node_modules/ws/` |
| Adom HTTPS proxy returns `error code: 502` with no body when the HMI's own dashboard endpoint fails | The Adom proxy strips bodies on 5xx (Cloudflare-style stock error page) | Have the HMI server return HTTP 200 with `{ok:false, error, hint}` instead of 5xx — see `octolux-app-debug` Symptom 6 |
| Dashboard section button (VNC / Deploy / Stream) does nothing | Different issue — Hydrogen browser session disconnected or server.js handler — see `octolux-app-debug` Symptoms 5 and 6 | Not in this skill |

## Things NOT to do

- **Don't blindly `open(ports[0])`** — always filter by VID. The kiosk policy returns the full set of `/dev/tty*` and most of them are platform UARTs that won't open.
- **Don't rely on `requestPort()`** — it's user-gesture-gated and shows a picker, and the picker is invisible to VNC, and the operator shouldn't have to tap anyway.
- **Don't walk `/proc/[0-9]*/net/tcp`** to find what's bound to a port. Namespace-wide view → `xargs kill` hits systemd PIDs.
- **Don't `pkill node`** on the kiosk. node-red, the screencast, and other helpers all run as `node`. Same rule that applies on the Adom container.
- **Don't restart Chromium without resetting screencast** — see `hmi-deploy-reset-vnc`. Same applies to CDP `Page.reload` and `Page.navigate`.
- **Don't return 5xx** for app errors that need an explanatory body — the Adom HTTPS proxy strips bodies on 5xx. Return HTTP 200 with `{ok: false, error, hint}`.
- **Don't assume udev metadata is populated** on the Yocto distro — `udevadm info /dev/ttyACM0` returns *"Inappropriate ioctl"*. Doesn't matter; Chromium reads VID/PID from the USB descriptor via `getInfo()`.

## Related skills

- `hmi-deploy-reset-vnc` (auto-memory) — the screencast-bounce rule. Triggered by ANY Chromium target change.
- `octolux-hmi-design` — visual design language (dark mode, accent colors, touch targets, Inter font) for the HTML body and CSS that wraps this skill's connect logic.
- `octolux-platform` — Octolux device-side reference (Weston, Chromium kiosk launcher, Apache config, paths).
- `octolux-deploy` — the wider HMI-deploy story; this skill's deploy sequence is a more reliable shell-equivalent of `octolux deploy push`.
- `octolux-app-debug` — when the issue is the Octolux Node server / dashboard, not the HMI/firmware pair.
- `rp2350-flash` — the SWD/flash side of the firmware lifecycle. This skill is the USB-runtime side; that one is the flash side.