web-serial-hmi — NTX HMI ↔ USB CDC firmware pattern

Scope. Building a Chromium-kiosk HMI on an NTX Octolux device that auto-connects to a USB-attached RP2040 or RP2350 target over CDC-ACM, with no Web Serial picker ever showing. End-to-end working pattern, not a per-customer recipe — instantiate it for any new project under ~/ntx-embedded/customers/<id>/.

Out of scope: Android USB Host (different stack); Web Bluetooth / WebHID; non-RP-series USB devices (mostly the same pattern, just substitute the VID); CAN-FD bridges; the SWD/flash side of the firmware lifecycle — that's covered by rp2350-flash and the target/rp2040.cfg openocd patterns and the local firmware/ build is just plain cmake + make.

When to use this skill

  • Starting a new customer HMI project that needs to control hardware over a USB-attached MCU.
  • Porting an Android-tablet HMI app to an Octolux Linux/Chromium device and you want the same "just plug it in" feel.
  • The Web Serial picker is showing — "No compatible devices found", "No port selected by user", or just appearing at all — and the project shouldn't have a picker.
  • requestPort() works once but the operator has to tap-and-pick every session and that's wrong for a kiosk.
  • The HMI page can't see what's happening because Chromium permission dialogs don't render in VNC.

Architecture

┌─────────────────────────────────┐      USB-C       ┌──────────────────────┐
│ NTX Octolux device (10.0.10.x)  │ ◄──────────────► │ RP2040 / RP2350 board│
│ ─────────────────────────────── │ CDC-ACM, 115200  │ ───────────────────  │
│ Chromium 109 kiosk @ localhost  │   8N1 line ASCII │ Pico SDK USB CDC     │
│  ├─ Web Serial API              │                  │  ├─ getchar polling  │
│  ├─ SerialAllowAllPortsForUrls  │                  │  ├─ ASCII line parser│
│  └─ getPorts() filtered by VID  │                  │  └─ GPIO drive       │
│ Apache at /                     │                  │                      │
│ /dev/ttyACM0 (root:root, 0666)  │                  │                      │
└─────────────────────────────────┘                  └──────────────────────┘
        ▲                                                       ▲
        │  SSH from container (deploy + udev + policy)          │  SWD via PiProbe (flash)
        │                                                       │
   Adom Docker container                                  see rp2350-flash
   ~/ntx-embedded/customers/<id>/{hmi,firmware}

Two physical links from the workcell to the target board:

  • USB-C between Octolux and target — runtime command path (this skill).
  • SWD between the workcell's PiProbe and the target — flash path (rp2350-flash).

The container is the build host for both: HMI HTML edits go SCP-ed to Octolux; firmware ELF is built locally and flashed via the PiProbe.

Required components

A working NTX HMI+firmware pair has five moving parts. Skip any one and connect won't happen. Pieces 3 and 4 are one-time setup per Octolux device; pieces 1, 2, 5 are per-project.

# Layer Where it lives Per-…
1 Firmware — USB CDC stdio, command parser, GPIO drive customers/<id>/firmware/src/main.c (in container) project
2 HMI HTML — connect logic + UI customers/<id>/hmi/index.html (in container, deployed to device) project
3 Chromium policySerialAllowAllPortsForUrls /etc/chromium/policies/managed/octolux-serial.json (on device) device
4 udev ruleMODE="0666" for VID 0x2e8a /etc/udev/rules.d/99-rp-cdc-acm.rules (on device) device
5 Deploy — SCP HMI + chromium restart + screencast bounce shell pattern run from container project

Quick start — new project, end to end

Run from the Adom container. Substitute <id> (e.g. acme-control), <device-ip> (e.g. 10.0.10.98), and your domain commands.

1. Scaffold

ID=acme-control
DEVICE_IP=10.0.10.98
DEVICE_MODEL='Octolux 15.6"'
VIEW_W=1920 VIEW_H=1080

mkdir -p ~/ntx-embedded/customers/$ID/{hmi,firmware/src}

cat > ~/ntx-embedded/customers/$ID/project.json <<EOF
{
  "id": "$ID",
  "name": "Acme Control",
  "customer": "Acme",
  "summary": "One-line description of what this HMI does.",
  "accent": "#e87a1e",
  "accentDeep": "#a64f10",
  "deviceModel": "$DEVICE_MODEL",
  "viewport": { "w": $VIEW_W, "h": $VIEW_H },
  "orientation": "landscape",
  "defaultIP": "$DEVICE_IP",
  "tagline": "one-line operator-facing tagline"
}
EOF

2. Drop in the firmware + HMI templates

Use the templates further down this doc verbatim, customizing only:

  • Firmware: FW_VERSION_STRING (the <NAME> FW x.y.z boot banner), the dispatch() command list, and your specific GPIO pin numbers / domain logic.
  • HMI: the visible controls (buttons, sliders, etc.), the brand colors from your project.json, the response-line handlers in handleResponse().

Keep the Web Serial connect logic and the visible debug strip unchanged — that's the point of this skill.

3. Build + flash the firmware (first time)

export PICO_SDK_PATH=$HOME/pico-sdk
cd ~/ntx-embedded/customers/$ID/firmware
mkdir -p build && cd build && cmake .. && make -j$(nproc)

# Flash via PiProbe over SWD — pick the cfg for your chip:
openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg \
  -c "adapter speed 5000; program ${ID//-/_}.elf verify reset exit"
# (or target/rp2350.cfg for an RP2350 — see rp2350-flash)

4. One-time device setup (Chromium policy + udev rule)

Skip if this Octolux device has been set up for Web Serial before. Verify by SSH'ing in: if /etc/chromium/policies/managed/octolux-serial.json and /etc/udev/rules.d/99-rp-cdc-acm.rules both exist, you're done.

ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
set -e
# Chromium policy: SerialAllowAllPortsForUrls turns getPorts() into a
# blanket enumeration for http://localhost; DefaultSerialGuardSetting=2
# also disables requestPort() pickers as a safety net.
mkdir -p /etc/chromium/policies/managed
cat > /etc/chromium/policies/managed/octolux-serial.json <<'POL'
{
  "SerialAllowAllPortsForUrls": ["http://localhost"],
  "DefaultSerialGuardSetting": 2
}
POL

# udev: /dev/ttyACM* is root-only by default on the Yocto distro, and
# the kiosk Chromium can't open it. MODE=0666 fixes that for any
# Raspberry Pi (RP2040/RP2350) USB CDC device.
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'UDEV'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
UDEV
udevadm control --reload-rules

# Re-process any ALREADY-plugged-in device so the new rule applies now.
# (Without this, the perms only apply to *future* plug events.)
BUSPORT=$(lsusb -t 2>/dev/null | awk '/Pico|2e8a/ {sub(/^[ \t]+/, ""); print}' | head -1)
echo "USB device tree row: ${BUSPORT:-(none seen)}"
# Cycle the parent USB device (substitute bus-port from `lsusb -t` if your
# topology differs; 4-1 is the common Octolux internal hub port).
echo 4-1 > /sys/bus/usb/drivers/usb/unbind 2>/dev/null || true
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind 2>/dev/null || true

systemctl restart octolux-chromium
REMOTE

# Per the [[hmi-deploy-reset-vnc]] rule: chromium restart leaves the
# screencast bound to a dead CDP target. Bounce the screencast on the
# device — see the Deploy sequence below for the exact commands.

5. Deploy the HMI to the device

See the "Standard deploy sequence" section below — it's the same shell pattern you'll run on every HMI edit.

6. Verify it auto-connected (CDP eval from container)

ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP

python3 - <<'PY'
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def evx(expr, await_=False):
    ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
        "params": {"expression": expr, "returnByValue": True,
                   "awaitPromise": await_}}))
    while True:
        r = json.loads(ws.recv())
        if r.get("id") == 1: return r["result"]["result"]["value"]
print("connected:", evx("state.connected"))
print("fwVersion:", evx("state.fwVersion"))
print("ports    :", evx("navigator.serial.getPorts().then(p => p.length)", await_=True))
PY

Expected once the firmware is running and the cable is plugged in:

connected: True
fwVersion: 1.0.0
ports    : 9          # 8 platform UARTs + your 1 RP-series CDC

If connected: True and fwVersion is populated → you're done; the HMI is talking to the MCU. If not, jump to the failure table.

Firmware template (RP2040 / RP2350 USB CDC)

firmware/CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)
set(PICO_BOARD pico CACHE STRING "")            # or "pico2" for RP2350
set(PICO_PLATFORM rp2040 CACHE STRING "")       # or "rp2350"
include($ENV{PICO_SDK_PATH}/external/pico_sdk_import.cmake)
project(<id_underscored> C CXX ASM)
pico_sdk_init()
add_executable(<id_underscored> src/main.c)
target_link_libraries(<id_underscored> pico_stdlib hardware_gpio)
pico_enable_stdio_usb(<id_underscored>  1)      # CDC-ACM on USB-C
pico_enable_stdio_uart(<id_underscored> 0)      # no UART — USB only
pico_add_extra_outputs(<id_underscored>)

firmware/src/main.c:

#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdarg.h>
#include "pico/stdlib.h"
#include "pico/stdio_usb.h"      // stdio_usb_connected()
#include "hardware/gpio.h"

#define FW_VERSION_STRING "<NAME> FW 1.0.0"
#define CMD_BUF_SIZE 96

// Status LEDs (wire your board's status LEDs to these or change as needed).
// LED_USB: solid ON while USB CDC is connected to a host.
// LED_ACT: 50 ms blink on every valid command.
// LED_ERR: solid ON when the last response was ERR; clears on next OK.
// LED_IDLE: solid ON when system is in its safe / inactive state.
#define LED_USB   9
#define LED_ACT  10
#define LED_ERR  16
#define LED_IDLE 18

static char cmd_buf[CMD_BUF_SIZE];
static int  cmd_len = 0;
static bool act_on = false;
static absolute_time_t act_off_at;

static void blink_activity(void) {
    gpio_put(LED_ACT, 1);
    act_on = true;
    act_off_at = make_timeout_time_ms(50);
}
static void say_ok(const char *fmt, ...) {
    va_list a; va_start(a, fmt);
    fputs("OK ", stdout); vprintf(fmt, a); fputs("\r\n", stdout);
    va_end(a);
    gpio_put(LED_ERR, 0);
    blink_activity();
}
static void say_err(const char *m) {
    printf("ERR %s\r\n", m);
    gpio_put(LED_ERR, 1);
    blink_activity();
}
static void to_upper(char *s) { for (; *s; s++) *s = (char)toupper((unsigned char)*s); }

// === Domain commands ====================================================
//
// Replace these with the commands your project actually needs. Keep
// the canonical built-ins (PING, VERSION, STATUS, HELP) — the HMI
// template's handshake assumes they exist.
//
static void dispatch(char *line) {
    char *tok = strtok(line, " \t");
    if (!tok) return;

    if (!strcmp(tok, "PING"))    { say_ok("PONG"); return; }
    if (!strcmp(tok, "VERSION")) { printf("%s\r\n", FW_VERSION_STRING); blink_activity(); return; }
    if (!strcmp(tok, "STATUS"))  { /* print one-line state snapshot */ blink_activity(); return; }
    if (!strcmp(tok, "HELP"))    { /* print "OK …" lines */ blink_activity(); return; }

    // your-domain command parsing here, e.g.:
    //   if (!strcmp(tok, "DO")) { … say_ok("DO %d", arg); return; }

    say_err("unknown command");
}

// === Boilerplate main =================================================

int main(void) {
    stdio_init_all();

    // Init your GPIOs here (LEDs + outputs). Drive outputs LOW first
    // so the system boots in a safe state regardless of host behavior.
    const uint leds[] = { LED_USB, LED_ACT, LED_ERR, LED_IDLE };
    for (size_t i = 0; i < sizeof(leds)/sizeof(leds[0]); i++) {
        gpio_init(leds[i]); gpio_set_dir(leds[i], GPIO_OUT); gpio_put(leds[i], 0);
    }
    gpio_put(LED_IDLE, 1);

    bool was_conn = false, banner_done = false;

    while (true) {
        bool conn = stdio_usb_connected();
        gpio_put(LED_USB, conn ? 1 : 0);
        if (conn && !was_conn) banner_done = false;
        was_conn = conn;

        // Boot banner on every CDC connect — the HMI parses this to
        // populate the FW version pill.
        if (conn && !banner_done) {
            sleep_ms(50);                     // give the host's reader a beat
            printf("%s\r\n", FW_VERSION_STRING);
            puts("TYPE 'HELP' FOR COMMANDS");
            banner_done = true;
        }

        // Drain available input into cmd_buf; dispatch on EOL.
        for (int budget = 0; budget < 32; budget++) {
            int c = getchar_timeout_us(0);
            if (c == PICO_ERROR_TIMEOUT) break;
            if (c == '\n' || c == '\r') {
                if (cmd_len > 0) {
                    cmd_buf[cmd_len] = '\0';
                    to_upper(cmd_buf);
                    dispatch(cmd_buf);
                    cmd_len = 0;
                }
            } else if (cmd_len < CMD_BUF_SIZE - 1) {
                cmd_buf[cmd_len++] = (char)c;
            } else { cmd_len = 0; say_err("command too long"); }
        }

        // Activity LED timeout.
        if (act_on && absolute_time_diff_us(get_absolute_time(), act_off_at) <= 0) {
            gpio_put(LED_ACT, 0); act_on = false;
        }
    }
}

Protocol conventions that work well

  • Newline-terminated ASCII — accept \n and \r\n. Case-insensitive (uppercase before dispatch).
  • One line in, exactly one line outOK …, ERR …, or a single-token status line. One-to-one keeps the HMI's send/receive trivial.
  • Standard built-ins always present:
    • PING → PONG (liveness)
    • VERSION → <NAME> FW x.y.z (capability detection + version pill)
    • STATUS → STATE … (one-line state snapshot)
    • HELP → multi-line OK … (operator self-discovery)
  • Boot banner on every CDC connect. Print <NAME> FW x.y.z\r\n first thing after stdio_usb_connected() flips true. The HMI matches /^([\w-]+) FW (\S+)/i to populate the version pill.
  • LED1 (USB) mirrors stdio_usb_connected() — instant visual confirmation that the host opened the port.
  • LED2 (activity) blinks 50 ms on every valid command — independent visual signal that the firmware parsed a command.
  • Default-safe on boot — all outputs LOW, critical actuators OFF, idle LED ON. Don't rely on the host to send ALL OFF first.

HMI template

<script>
"use strict";

const state = {
  port: null, writer: null, reader: null,
  connected: false, fwVersion: null,
  autoConnectTimer: null,
};

// ─── Visible debug strip ──────────────────────────────────────────────
// Mirror every UI event + Web Serial step into the page DOM, because
// Chromium permission dialogs (the Web Serial picker, geo prompts,
// etc.) render above the page surface and DON'T appear in the
// screencast / VNC feed. Without this strip, you'd be blind to what's
// happening on the kiosk from your dashboard.
function dbg(line) {
  const strip = document.getElementById('debugStrip');
  if (!strip) return console.log(line);
  const d = document.createElement('div');
  d.textContent = new Date().toISOString().slice(11,23) + '  ' + line;
  strip.appendChild(d);
  while (strip.children.length > 5) strip.removeChild(strip.firstChild);
}

// ─── Line-splitter transformer for the reader stream ──────────────────
function lineSplitter() {
  let buf = '';
  return {
    transform(chunk, ctrl) {
      buf += chunk;
      let i;
      while ((i = buf.search(/\r?\n/)) >= 0) {
        ctrl.enqueue(buf.slice(0, i));
        buf = buf.slice(i + (buf[i] === '\r' ? 2 : 1));
      }
    },
  };
}

async function openExisting(port) {
  await port.open({
    baudRate: 115200, dataBits: 8, stopBits: 1, parity: 'none',
    bufferSize: 256, flowControl: 'none',
  });
  state.port = port;

  const enc = new TextEncoderStream();
  enc.readable.pipeTo(port.writable);
  state.writer = enc.writable.getWriter();

  const dec = new TextDecoderStream();
  port.readable.pipeTo(dec.writable);
  state.reader = dec.readable
    .pipeThrough(new TransformStream(lineSplitter()))
    .getReader();

  state.connected = true;
  readLoop();
  handshake();
}

async function tryAutoConnect() {
  if (state.connected || !('serial' in navigator)) return state.connected;
  const ports = await navigator.serial.getPorts();
  // The kiosk policy returns the FULL set of /dev/tty* devices the
  // kernel sees — typically ~8 platform UARTs (empty getInfo()) plus
  // your real USB CDC. Always filter; never open ports[0] blindly.
  // 0x2e8a is the Raspberry Pi VID (RP2040 + RP2350). Substitute your
  // chip's VID if you use a non-RP target.
  const rp     = ports.filter(p => (p.getInfo() || {}).usbVendorId === 0x2e8a);
  const anyUsb = ports.filter(p => typeof (p.getInfo() || {}).usbVendorId === 'number');
  const candidates = rp.length ? rp : anyUsb;
  dbg(`getPorts() → ${ports.length} total, ${rp.length} RP-series`);
  for (const port of candidates) {
    try {
      await openExisting(port);
      const info = port.getInfo();
      dbg(`auto-connected to VID:${(info.usbVendorId||0).toString(16)} PID:${(info.usbProductId||0).toString(16)}`);
      return true;
    } catch (e) {
      dbg('  open failed on one candidate: ' + e.message);
    }
  }
  return false;
}

window.addEventListener('load', async () => {
  // Install global capture-phase event listeners so we can see every
  // tap on the page, even ones that don't land on a button.
  ['pointerdown','pointerup','touchstart','touchend',
   'mousedown','mouseup','click'].forEach(k => {
    document.addEventListener(k, e => {
      const tgt = e.target ? (e.target.id || e.target.tagName) : '?';
      dbg(`${k}  (${(e.clientX||0)|0},${(e.clientY||0)|0}) → ${tgt}  trusted=${e.isTrusted}`);
    }, /* useCapture */ true);
  });
  dbg('debug listeners installed');

  if (!(await tryAutoConnect())) {
    // Keep polling — if the cable is plugged in *after* the page loads,
    // the next tick picks it up. Cheap (one promise every 2 s).
    state.autoConnectTimer = setInterval(async () => {
      if (await tryAutoConnect() && state.autoConnectTimer) {
        clearInterval(state.autoConnectTimer);
        state.autoConnectTimer = null;
      }
    }, 2000);
  }
});

async function readLoop() {
  while (state.reader) {
    try {
      const { value, done } = await state.reader.read();
      if (done) { state.connected = false; break; }
      if (value) handleResponse(value.trim());
    } catch { state.connected = false; break; }
  }
}

async function handshake() {
  await send('PING');
  await send('VERSION');
  await send('STATUS');
}

async function send(cmd) {
  if (!state.writer) return;
  dbg('→ ' + cmd);
  await state.writer.write(cmd + '\n');
}

function handleResponse(line) {
  dbg('← ' + line);
  const m = line.match(/^([\w-]+) FW (\S+)/i);
  if (m) state.fwVersion = m[2];
  // Your domain handlers here (parse STATUS reply, react to OK/ERR, etc.)
}
</script>

The HMI's HTML body and CSS are entirely up to your project — see octolux-hmi-design for the brand-language guide (dark mode, accent colors, touch-target sizes, Inter font). The only DOM element this template requires is <div id="debugStrip"></div> somewhere visible during development. You can hide it (or remove the global tap listeners) after the project ships.

One-time Chromium policy — the single biggest enabler

Without SerialAllowAllPortsForUrls, requestPort() shows a picker, the picker isn't visible through VNC, and the kiosk is unreliable. With it, getPorts() returns every kernel serial device with no user gesture.

{
  "SerialAllowAllPortsForUrls": ["http://localhost"],
  "DefaultSerialGuardSetting": 2
}
  • SerialAllowAllPortsForUrls: ["http://localhost"] — pages served from http://localhost (which is where the Octolux's Apache serves the HMI) get blanket permission to enumerate and open serial ports. getPorts() now returns the full list.
  • DefaultSerialGuardSetting: 22 means "Do not allow any site to request access". Belt-and-suspenders: if your code accidentally calls requestPort(), the picker is a no-op instead of presenting UI nobody can see.

Path: /etc/chromium/policies/managed/octolux-serial.json. Restart Chromium after install (systemctl restart octolux-chromium). After restart, you must also bounce the screencast (see hmi-deploy-reset-vnc).

Verify the policy is live via CDP eval:

navigator.serial.getPorts().then(p => p.length)
// Expected: a positive integer (typically 9 on Octolux 15.6") when an RP
// device is plugged in. Zero means the policy didn't load — re-check
// the file path and the chromium restart.

One-time udev rule

The Octolux Yocto distro ships /dev/ttyACM0 as -rw-r--r-- root:root. The kiosk Chromium can't open it. Fix with a rule that targets the RP-series VID:

cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'EOF'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
EOF
udevadm control --reload-rules

For an already-plugged device, the rule doesn't retroactively re-permission. Either physically unplug+replug, or cycle the USB driver from a shell:

echo 4-1 > /sys/bus/usb/drivers/usb/unbind
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind

(4-1 is the typical Octolux internal hub port; substitute from lsusb -t if your topology differs.)

Note: the Yocto distro lacks the usb_id udev helper, so udevadm info /dev/ttyACM0 returns "Unknown device: Inappropriate ioctl". That's fine. Chromium reads getInfo().usbVendorId from the underlying USB descriptor, not from udev's metadata database. You only need the permission rule, not metadata population.

Standard deploy sequence

Every HMI edit goes through this. Don't skip the screencast bounce — see hmi-deploy-reset-vnc.

ID=acme-control
DEVICE_IP=10.0.10.98

# 1. Push the HTML
scp ~/ntx-embedded/customers/$ID/hmi/index.html \
    root@$DEVICE_IP:/usr/share/apache2/default-site/htdocs/index.html

# 2. Reload the kiosk — restart chromium pulls the new HTML AND
#    reloads any new chromium policy / udev rule deployed alongside.
ssh root@$DEVICE_IP 'systemctl restart octolux-chromium'

# 3. Wait for the kiosk to come back
sleep 10

# 4. Bounce the screencast — chromium restart killed its CDP target
#    binding and the screencast won't auto-reattach. Without this,
#    the dashboard VNC tile freezes on a pre-deploy frame.
#    NOTE: busybox `ps aux` truncates cmdlines so `ps aux | grep
#    screencast` returns NOTHING even when it's running. Walk /proc
#    instead. And NEVER walk /proc/PID/net/tcp to find port owners —
#    those views are namespace-wide and you will mass-kill systemd.
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
for d in /proc/[0-9]*; do
  pid=${d##*/}
  grep -q 'screencast' "$d/cmdline" 2>/dev/null && kill -9 "$pid" 2>/dev/null
done
sleep 2
cd /tmp && nohup node screencast.js --quality 80 --pipeline 4 > /tmp/screencast.log 2>&1 &
REMOTE

# 5. (optional) confirm the dashboard VNC is alive — successive snaps
#    should differ unless the page is genuinely static.
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/a.jpg -w "%{size_download}\n"
sleep 2
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/b.jpg -w "%{size_download}\n"

If /tmp/screencast.js or /tmp/node_modules/ws/ ever go missing on the device (they live in tmpfs and can be wiped by a bad kill spree), SCP them back from the container:

scp ~/ntx-embedded/apps/octolux/ui/screencast/screencast.js \
    root@$DEVICE_IP:/tmp/screencast.js
cd ~/ntx-embedded/apps/octolux/node_modules && tar cz ws | \
  ssh root@$DEVICE_IP 'mkdir -p /tmp/node_modules && cd /tmp/node_modules && tar xz'

Live diagnostics — CDP from the container

The Octolux kiosk exposes Chrome DevTools at localhost:9222. Tunnel it back:

ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP

# install python websocket client once
pip install --user --break-system-packages websocket-client

Then any time you want to inspect the live page state:

import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def cdp(method, params=None, _id=[0]):
    _id[0] += 1
    ws.send(json.dumps({"id": _id[0], "method": method, "params": params or {}}))
    while True:
        r = json.loads(ws.recv())
        if r.get("id") == _id[0]: return r
def evx(expr, await_=False):
    return cdp("Runtime.evaluate", {"expression": expr, "returnByValue": True, "awaitPromise": await_})["result"]["result"]["value"]

# Useful queries:
evx("state.connected")
evx("state.fwVersion")
evx("navigator.serial.getPorts().then(p => p.length)", await_=True)
evx("Array.from(document.querySelectorAll('#debugStrip > div')).map(d=>d.textContent).join('\\n')")

# Force a reload (and remember: bounce screencast afterwards!):
cdp("Page.reload", {"ignoreCache": True})

Hit-test if a tap doesn't fire the button

const b = document.getElementById('btnConnect');
const r = b.getBoundingClientRect();
const hit = document.elementFromPoint(r.left + r.width/2, r.top + r.height/2);
JSON.stringify({ same: hit === b, hit_id: hit?.id, hit_tag: hit?.tagName });

If same is false, something is covering the button.

Failure-mode reference

Every row in this table corresponds to a real failure observed during this pattern's development. Use it as the diagnostic flow.

Symptom Root cause Fix
Picker shows "No compatible devices found" Chromium policy not in place AND/OR /dev/ttyACM* not world-readable Install both the policy and the udev rule; restart chromium; bounce screencast
Picker shows the device but every connect ends in NotFoundError: No port selected by user User dismissed the picker, or requestPort() was called from a non-user-gesture context (programmatic CDP .click() triggers this) Don't use requestPort() at all — use the getPorts() auto-connect pattern in the HMI template
getPorts() returns ports but port.open() throws Failed to execute 'open' on 'SerialPort': Failed to open serial port You opened ports[0]; it's a platform UART without a backing device Filter by getInfo().usbVendorId === 0x2e8a before opening
getPorts().length === 0 even though lsusb shows the device on the kernel side Policy file missing, malformed, or wrong path; OR Chromium wasn't restarted after the policy install; OR /dev/ttyACM0 perms are still root-only Re-check /etc/chromium/policies/managed/octolux-serial.json is valid JSON; restart chromium; ls -la /dev/ttyACM0 should be 0666
Tap on a button does nothing — no visible page reaction Touch input might not be reaching the page; check via the debug strip (it should log pointerdown / click). If those don't appear → input layer (Weston/touchscreen) issue. If they do appear but connect() called doesn't → the click hit something covering the button — use elementFromPoint Diagnose via debug strip + hit-test
The debug strip shows connect() called → requestPort()... and hangs — then eventually fails "No port selected by user" The HMI still has requestPort() somewhere Replace with getPorts() + VID filter (HMI template)
Page reloaded via CDP Page.reload — VNC dashboard tile froze on the pre-reload frame The screencast bound to the old CDP target ID; reload creates a new one; the binding is dead Bounce screencast (Standard deploy sequence step 4). Applies to ANY Chromium target change — restart, reload, navigate
octolux deploy push times out Octolux server's deploy API is flaky on this build Use the SCP + restart bash sequence directly (it's what the API does internally anyway)
ps aux | grep screencast returns nothing but screencast IS running and bound to port 8085 Busybox ps aux truncates cmdlines Walk /proc/[0-9]*/cmdline to find PIDs by argv match
Killed a system PID and lost SSH / had to recover the device You walked /proc/[0-9]*/net/tcp to find owners of port 8085 — those views are namespace-wide and match every PID NEVER identify port owners via /proc/PID/net/tcp on this distro. Use /proc/PID/cmdline matching instead
/tmp/screencast.js or /tmp/node_modules/ws/ missing after a bad kill / reboot They live in tmpfs and the original placement path doesn't auto-restore SCP from ~/ntx-embedded/apps/octolux/ui/screencast/screencast.js and tar+SCP ~/ntx-embedded/apps/octolux/node_modules/ws/
Adom HTTPS proxy returns error code: 502 with no body when the HMI's own dashboard endpoint fails The Adom proxy strips bodies on 5xx (Cloudflare-style stock error page) Have the HMI server return HTTP 200 with {ok:false, error, hint} instead of 5xx — see octolux-app-debug Symptom 6
Dashboard section button (VNC / Deploy / Stream) does nothing Different issue — Hydrogen browser session disconnected or server.js handler — see octolux-app-debug Symptoms 5 and 6 Not in this skill

Things NOT to do

  • Don't blindly open(ports[0]) — always filter by VID. The kiosk policy returns the full set of /dev/tty* and most of them are platform UARTs that won't open.
  • Don't rely on requestPort() — it's user-gesture-gated and shows a picker, and the picker is invisible to VNC, and the operator shouldn't have to tap anyway.
  • Don't walk /proc/[0-9]*/net/tcp to find what's bound to a port. Namespace-wide view → xargs kill hits systemd PIDs.
  • Don't pkill node on the kiosk. node-red, the screencast, and other helpers all run as node. Same rule that applies on the Adom container.
  • Don't restart Chromium without resetting screencast — see hmi-deploy-reset-vnc. Same applies to CDP Page.reload and Page.navigate.
  • Don't return 5xx for app errors that need an explanatory body — the Adom HTTPS proxy strips bodies on 5xx. Return HTTP 200 with {ok: false, error, hint}.
  • Don't assume udev metadata is populated on the Yocto distro — udevadm info /dev/ttyACM0 returns "Inappropriate ioctl". Doesn't matter; Chromium reads VID/PID from the USB descriptor via getInfo().
  • hmi-deploy-reset-vnc (auto-memory) — the screencast-bounce rule. Triggered by ANY Chromium target change.
  • octolux-hmi-design — visual design language (dark mode, accent colors, touch targets, Inter font) for the HTML body and CSS that wraps this skill's connect logic.
  • octolux-platform — Octolux device-side reference (Weston, Chromium kiosk launcher, Apache config, paths).
  • octolux-deploy — the wider HMI-deploy story; this skill's deploy sequence is a more reliable shell-equivalent of octolux deploy push.
  • octolux-app-debug — when the issue is the Octolux Node server / dashboard, not the HMI/firmware pair.
  • rp2350-flash — the SWD/flash side of the firmware lifecycle. This skill is the USB-runtime side; that one is the flash side.