Web Serial HMI + USB CDC Firmware (Octolux ↔ RP-series)
Public Unreviewedby Caleb
Build a paired NTX HMI (Chromium kiosk on Octolux) + firmware (USB CDC on RP2040/RP2350) that auto-connects over USB with no permission picker. End-to-end recipe: Pico SDK firmware template, HMI Web Serial via getPorts() + VID filter, Chromium SerialAllowAllPortsForUrls policy, udev MODE=0666 for /dev/ttyACM*, SCP + chromium-restart + screencast-bounce deploy, CDP verification, visible-debug-strip diagnostics, and a 12-row failure-mode table.
web-serial-hmi — NTX HMI ↔ USB CDC firmware pattern
Scope. Building a Chromium-kiosk HMI on an NTX Octolux device that auto-connects to a USB-attached RP2040 or RP2350 target over CDC-ACM, with no Web Serial picker ever showing. End-to-end working pattern, not a per-customer recipe — instantiate it for any new project under
~/ntx-embedded/customers/<id>/.Out of scope: Android USB Host (different stack); Web Bluetooth / WebHID; non-RP-series USB devices (mostly the same pattern, just substitute the VID); CAN-FD bridges; the SWD/flash side of the firmware lifecycle — that's covered by
rp2350-flashand thetarget/rp2040.cfgopenocd patterns and the localfirmware/build is just plaincmake + make.
When to use this skill
- Starting a new customer HMI project that needs to control hardware over a USB-attached MCU.
- Porting an Android-tablet HMI app to an Octolux Linux/Chromium device and you want the same "just plug it in" feel.
- The Web Serial picker is showing — "No compatible devices found", "No port selected by user", or just appearing at all — and the project shouldn't have a picker.
requestPort()works once but the operator has to tap-and-pick every session and that's wrong for a kiosk.- The HMI page can't see what's happening because Chromium permission dialogs don't render in VNC.
Architecture
┌─────────────────────────────────┐ USB-C ┌──────────────────────┐
│ NTX Octolux device (10.0.10.x) │ ◄──────────────► │ RP2040 / RP2350 board│
│ ─────────────────────────────── │ CDC-ACM, 115200 │ ─────────────────── │
│ Chromium 109 kiosk @ localhost │ 8N1 line ASCII │ Pico SDK USB CDC │
│ ├─ Web Serial API │ │ ├─ getchar polling │
│ ├─ SerialAllowAllPortsForUrls │ │ ├─ ASCII line parser│
│ └─ getPorts() filtered by VID │ │ └─ GPIO drive │
│ Apache at / │ │ │
│ /dev/ttyACM0 (root:root, 0666) │ │ │
└─────────────────────────────────┘ └──────────────────────┘
▲ ▲
│ SSH from container (deploy + udev + policy) │ SWD via PiProbe (flash)
│ │
Adom Docker container see rp2350-flash
~/ntx-embedded/customers/<id>/{hmi,firmware}
Two physical links from the workcell to the target board:
- USB-C between Octolux and target — runtime command path (this skill).
- SWD between the workcell's PiProbe and the target — flash path (
rp2350-flash).
The container is the build host for both: HMI HTML edits go SCP-ed to Octolux; firmware ELF is built locally and flashed via the PiProbe.
Required components
A working NTX HMI+firmware pair has five moving parts. Skip any one and connect won't happen. Pieces 3 and 4 are one-time setup per Octolux device; pieces 1, 2, 5 are per-project.
| # | Layer | Where it lives | Per-… |
|---|---|---|---|
| 1 | Firmware — USB CDC stdio, command parser, GPIO drive | customers/<id>/firmware/src/main.c (in container) |
project |
| 2 | HMI HTML — connect logic + UI | customers/<id>/hmi/index.html (in container, deployed to device) |
project |
| 3 | Chromium policy — SerialAllowAllPortsForUrls |
/etc/chromium/policies/managed/octolux-serial.json (on device) |
device |
| 4 | udev rule — MODE="0666" for VID 0x2e8a |
/etc/udev/rules.d/99-rp-cdc-acm.rules (on device) |
device |
| 5 | Deploy — SCP HMI + chromium restart + screencast bounce | shell pattern run from container | project |
Quick start — new project, end to end
Run from the Adom container. Substitute <id> (e.g. acme-control), <device-ip> (e.g. 10.0.10.98), and your domain commands.
1. Scaffold
ID=acme-control
DEVICE_IP=10.0.10.98
DEVICE_MODEL='Octolux 15.6"'
VIEW_W=1920 VIEW_H=1080
mkdir -p ~/ntx-embedded/customers/$ID/{hmi,firmware/src}
cat > ~/ntx-embedded/customers/$ID/project.json <<EOF
{
"id": "$ID",
"name": "Acme Control",
"customer": "Acme",
"summary": "One-line description of what this HMI does.",
"accent": "#e87a1e",
"accentDeep": "#a64f10",
"deviceModel": "$DEVICE_MODEL",
"viewport": { "w": $VIEW_W, "h": $VIEW_H },
"orientation": "landscape",
"defaultIP": "$DEVICE_IP",
"tagline": "one-line operator-facing tagline"
}
EOF
2. Drop in the firmware + HMI templates
Use the templates further down this doc verbatim, customizing only:
- Firmware:
FW_VERSION_STRING(the<NAME> FW x.y.zboot banner), thedispatch()command list, and your specific GPIO pin numbers / domain logic. - HMI: the visible controls (buttons, sliders, etc.), the brand colors from your
project.json, the response-line handlers inhandleResponse().
Keep the Web Serial connect logic and the visible debug strip unchanged — that's the point of this skill.
3. Build + flash the firmware (first time)
export PICO_SDK_PATH=$HOME/pico-sdk
cd ~/ntx-embedded/customers/$ID/firmware
mkdir -p build && cd build && cmake .. && make -j$(nproc)
# Flash via PiProbe over SWD — pick the cfg for your chip:
openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg \
-c "adapter speed 5000; program ${ID//-/_}.elf verify reset exit"
# (or target/rp2350.cfg for an RP2350 — see rp2350-flash)
4. One-time device setup (Chromium policy + udev rule)
Skip if this Octolux device has been set up for Web Serial before. Verify by SSH'ing in: if /etc/chromium/policies/managed/octolux-serial.json and /etc/udev/rules.d/99-rp-cdc-acm.rules both exist, you're done.
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
set -e
# Chromium policy: SerialAllowAllPortsForUrls turns getPorts() into a
# blanket enumeration for http://localhost; DefaultSerialGuardSetting=2
# also disables requestPort() pickers as a safety net.
mkdir -p /etc/chromium/policies/managed
cat > /etc/chromium/policies/managed/octolux-serial.json <<'POL'
{
"SerialAllowAllPortsForUrls": ["http://localhost"],
"DefaultSerialGuardSetting": 2
}
POL
# udev: /dev/ttyACM* is root-only by default on the Yocto distro, and
# the kiosk Chromium can't open it. MODE=0666 fixes that for any
# Raspberry Pi (RP2040/RP2350) USB CDC device.
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'UDEV'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
UDEV
udevadm control --reload-rules
# Re-process any ALREADY-plugged-in device so the new rule applies now.
# (Without this, the perms only apply to *future* plug events.)
BUSPORT=$(lsusb -t 2>/dev/null | awk '/Pico|2e8a/ {sub(/^[ \t]+/, ""); print}' | head -1)
echo "USB device tree row: ${BUSPORT:-(none seen)}"
# Cycle the parent USB device (substitute bus-port from `lsusb -t` if your
# topology differs; 4-1 is the common Octolux internal hub port).
echo 4-1 > /sys/bus/usb/drivers/usb/unbind 2>/dev/null || true
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind 2>/dev/null || true
systemctl restart octolux-chromium
REMOTE
# Per the [[hmi-deploy-reset-vnc]] rule: chromium restart leaves the
# screencast bound to a dead CDP target. Bounce the screencast on the
# device — see the Deploy sequence below for the exact commands.
5. Deploy the HMI to the device
See the "Standard deploy sequence" section below — it's the same shell pattern you'll run on every HMI edit.
6. Verify it auto-connected (CDP eval from container)
ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP
python3 - <<'PY'
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def evx(expr, await_=False):
ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
"params": {"expression": expr, "returnByValue": True,
"awaitPromise": await_}}))
while True:
r = json.loads(ws.recv())
if r.get("id") == 1: return r["result"]["result"]["value"]
print("connected:", evx("state.connected"))
print("fwVersion:", evx("state.fwVersion"))
print("ports :", evx("navigator.serial.getPorts().then(p => p.length)", await_=True))
PY
Expected once the firmware is running and the cable is plugged in:
connected: True
fwVersion: 1.0.0
ports : 9 # 8 platform UARTs + your 1 RP-series CDC
If connected: True and fwVersion is populated → you're done; the HMI is talking to the MCU. If not, jump to the failure table.
Firmware template (RP2040 / RP2350 USB CDC)
firmware/CMakeLists.txt:
cmake_minimum_required(VERSION 3.13)
set(PICO_BOARD pico CACHE STRING "") # or "pico2" for RP2350
set(PICO_PLATFORM rp2040 CACHE STRING "") # or "rp2350"
include($ENV{PICO_SDK_PATH}/external/pico_sdk_import.cmake)
project(<id_underscored> C CXX ASM)
pico_sdk_init()
add_executable(<id_underscored> src/main.c)
target_link_libraries(<id_underscored> pico_stdlib hardware_gpio)
pico_enable_stdio_usb(<id_underscored> 1) # CDC-ACM on USB-C
pico_enable_stdio_uart(<id_underscored> 0) # no UART — USB only
pico_add_extra_outputs(<id_underscored>)
firmware/src/main.c:
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdarg.h>
#include "pico/stdlib.h"
#include "pico/stdio_usb.h" // stdio_usb_connected()
#include "hardware/gpio.h"
#define FW_VERSION_STRING "<NAME> FW 1.0.0"
#define CMD_BUF_SIZE 96
// Status LEDs (wire your board's status LEDs to these or change as needed).
// LED_USB: solid ON while USB CDC is connected to a host.
// LED_ACT: 50 ms blink on every valid command.
// LED_ERR: solid ON when the last response was ERR; clears on next OK.
// LED_IDLE: solid ON when system is in its safe / inactive state.
#define LED_USB 9
#define LED_ACT 10
#define LED_ERR 16
#define LED_IDLE 18
static char cmd_buf[CMD_BUF_SIZE];
static int cmd_len = 0;
static bool act_on = false;
static absolute_time_t act_off_at;
static void blink_activity(void) {
gpio_put(LED_ACT, 1);
act_on = true;
act_off_at = make_timeout_time_ms(50);
}
static void say_ok(const char *fmt, ...) {
va_list a; va_start(a, fmt);
fputs("OK ", stdout); vprintf(fmt, a); fputs("\r\n", stdout);
va_end(a);
gpio_put(LED_ERR, 0);
blink_activity();
}
static void say_err(const char *m) {
printf("ERR %s\r\n", m);
gpio_put(LED_ERR, 1);
blink_activity();
}
static void to_upper(char *s) { for (; *s; s++) *s = (char)toupper((unsigned char)*s); }
// === Domain commands ====================================================
//
// Replace these with the commands your project actually needs. Keep
// the canonical built-ins (PING, VERSION, STATUS, HELP) — the HMI
// template's handshake assumes they exist.
//
static void dispatch(char *line) {
char *tok = strtok(line, " \t");
if (!tok) return;
if (!strcmp(tok, "PING")) { say_ok("PONG"); return; }
if (!strcmp(tok, "VERSION")) { printf("%s\r\n", FW_VERSION_STRING); blink_activity(); return; }
if (!strcmp(tok, "STATUS")) { /* print one-line state snapshot */ blink_activity(); return; }
if (!strcmp(tok, "HELP")) { /* print "OK …" lines */ blink_activity(); return; }
// your-domain command parsing here, e.g.:
// if (!strcmp(tok, "DO")) { … say_ok("DO %d", arg); return; }
say_err("unknown command");
}
// === Boilerplate main =================================================
int main(void) {
stdio_init_all();
// Init your GPIOs here (LEDs + outputs). Drive outputs LOW first
// so the system boots in a safe state regardless of host behavior.
const uint leds[] = { LED_USB, LED_ACT, LED_ERR, LED_IDLE };
for (size_t i = 0; i < sizeof(leds)/sizeof(leds[0]); i++) {
gpio_init(leds[i]); gpio_set_dir(leds[i], GPIO_OUT); gpio_put(leds[i], 0);
}
gpio_put(LED_IDLE, 1);
bool was_conn = false, banner_done = false;
while (true) {
bool conn = stdio_usb_connected();
gpio_put(LED_USB, conn ? 1 : 0);
if (conn && !was_conn) banner_done = false;
was_conn = conn;
// Boot banner on every CDC connect — the HMI parses this to
// populate the FW version pill.
if (conn && !banner_done) {
sleep_ms(50); // give the host's reader a beat
printf("%s\r\n", FW_VERSION_STRING);
puts("TYPE 'HELP' FOR COMMANDS");
banner_done = true;
}
// Drain available input into cmd_buf; dispatch on EOL.
for (int budget = 0; budget < 32; budget++) {
int c = getchar_timeout_us(0);
if (c == PICO_ERROR_TIMEOUT) break;
if (c == '\n' || c == '\r') {
if (cmd_len > 0) {
cmd_buf[cmd_len] = '\0';
to_upper(cmd_buf);
dispatch(cmd_buf);
cmd_len = 0;
}
} else if (cmd_len < CMD_BUF_SIZE - 1) {
cmd_buf[cmd_len++] = (char)c;
} else { cmd_len = 0; say_err("command too long"); }
}
// Activity LED timeout.
if (act_on && absolute_time_diff_us(get_absolute_time(), act_off_at) <= 0) {
gpio_put(LED_ACT, 0); act_on = false;
}
}
}
Protocol conventions that work well
- Newline-terminated ASCII — accept
\nand\r\n. Case-insensitive (uppercase before dispatch). - One line in, exactly one line out —
OK …,ERR …, or a single-token status line. One-to-one keeps the HMI's send/receive trivial. - Standard built-ins always present:
PING → PONG(liveness)VERSION → <NAME> FW x.y.z(capability detection + version pill)STATUS → STATE …(one-line state snapshot)HELP → multi-line OK …(operator self-discovery)
- Boot banner on every CDC connect. Print
<NAME> FW x.y.z\r\nfirst thing afterstdio_usb_connected()flips true. The HMI matches/^([\w-]+) FW (\S+)/ito populate the version pill. - LED1 (USB) mirrors
stdio_usb_connected()— instant visual confirmation that the host opened the port. - LED2 (activity) blinks 50 ms on every valid command — independent visual signal that the firmware parsed a command.
- Default-safe on boot — all outputs LOW, critical actuators OFF, idle LED ON. Don't rely on the host to send
ALL OFFfirst.
HMI template
<script>
"use strict";
const state = {
port: null, writer: null, reader: null,
connected: false, fwVersion: null,
autoConnectTimer: null,
};
// ─── Visible debug strip ──────────────────────────────────────────────
// Mirror every UI event + Web Serial step into the page DOM, because
// Chromium permission dialogs (the Web Serial picker, geo prompts,
// etc.) render above the page surface and DON'T appear in the
// screencast / VNC feed. Without this strip, you'd be blind to what's
// happening on the kiosk from your dashboard.
function dbg(line) {
const strip = document.getElementById('debugStrip');
if (!strip) return console.log(line);
const d = document.createElement('div');
d.textContent = new Date().toISOString().slice(11,23) + ' ' + line;
strip.appendChild(d);
while (strip.children.length > 5) strip.removeChild(strip.firstChild);
}
// ─── Line-splitter transformer for the reader stream ──────────────────
function lineSplitter() {
let buf = '';
return {
transform(chunk, ctrl) {
buf += chunk;
let i;
while ((i = buf.search(/\r?\n/)) >= 0) {
ctrl.enqueue(buf.slice(0, i));
buf = buf.slice(i + (buf[i] === '\r' ? 2 : 1));
}
},
};
}
async function openExisting(port) {
await port.open({
baudRate: 115200, dataBits: 8, stopBits: 1, parity: 'none',
bufferSize: 256, flowControl: 'none',
});
state.port = port;
const enc = new TextEncoderStream();
enc.readable.pipeTo(port.writable);
state.writer = enc.writable.getWriter();
const dec = new TextDecoderStream();
port.readable.pipeTo(dec.writable);
state.reader = dec.readable
.pipeThrough(new TransformStream(lineSplitter()))
.getReader();
state.connected = true;
readLoop();
handshake();
}
async function tryAutoConnect() {
if (state.connected || !('serial' in navigator)) return state.connected;
const ports = await navigator.serial.getPorts();
// The kiosk policy returns the FULL set of /dev/tty* devices the
// kernel sees — typically ~8 platform UARTs (empty getInfo()) plus
// your real USB CDC. Always filter; never open ports[0] blindly.
// 0x2e8a is the Raspberry Pi VID (RP2040 + RP2350). Substitute your
// chip's VID if you use a non-RP target.
const rp = ports.filter(p => (p.getInfo() || {}).usbVendorId === 0x2e8a);
const anyUsb = ports.filter(p => typeof (p.getInfo() || {}).usbVendorId === 'number');
const candidates = rp.length ? rp : anyUsb;
dbg(`getPorts() → ${ports.length} total, ${rp.length} RP-series`);
for (const port of candidates) {
try {
await openExisting(port);
const info = port.getInfo();
dbg(`auto-connected to VID:${(info.usbVendorId||0).toString(16)} PID:${(info.usbProductId||0).toString(16)}`);
return true;
} catch (e) {
dbg(' open failed on one candidate: ' + e.message);
}
}
return false;
}
window.addEventListener('load', async () => {
// Install global capture-phase event listeners so we can see every
// tap on the page, even ones that don't land on a button.
['pointerdown','pointerup','touchstart','touchend',
'mousedown','mouseup','click'].forEach(k => {
document.addEventListener(k, e => {
const tgt = e.target ? (e.target.id || e.target.tagName) : '?';
dbg(`${k} (${(e.clientX||0)|0},${(e.clientY||0)|0}) → ${tgt} trusted=${e.isTrusted}`);
}, /* useCapture */ true);
});
dbg('debug listeners installed');
if (!(await tryAutoConnect())) {
// Keep polling — if the cable is plugged in *after* the page loads,
// the next tick picks it up. Cheap (one promise every 2 s).
state.autoConnectTimer = setInterval(async () => {
if (await tryAutoConnect() && state.autoConnectTimer) {
clearInterval(state.autoConnectTimer);
state.autoConnectTimer = null;
}
}, 2000);
}
});
async function readLoop() {
while (state.reader) {
try {
const { value, done } = await state.reader.read();
if (done) { state.connected = false; break; }
if (value) handleResponse(value.trim());
} catch { state.connected = false; break; }
}
}
async function handshake() {
await send('PING');
await send('VERSION');
await send('STATUS');
}
async function send(cmd) {
if (!state.writer) return;
dbg('→ ' + cmd);
await state.writer.write(cmd + '\n');
}
function handleResponse(line) {
dbg('← ' + line);
const m = line.match(/^([\w-]+) FW (\S+)/i);
if (m) state.fwVersion = m[2];
// Your domain handlers here (parse STATUS reply, react to OK/ERR, etc.)
}
</script>
The HMI's HTML body and CSS are entirely up to your project — see octolux-hmi-design for the brand-language guide (dark mode, accent colors, touch-target sizes, Inter font). The only DOM element this template requires is <div id="debugStrip"></div> somewhere visible during development. You can hide it (or remove the global tap listeners) after the project ships.
One-time Chromium policy — the single biggest enabler
Without
SerialAllowAllPortsForUrls,requestPort()shows a picker, the picker isn't visible through VNC, and the kiosk is unreliable. With it,getPorts()returns every kernel serial device with no user gesture.
{
"SerialAllowAllPortsForUrls": ["http://localhost"],
"DefaultSerialGuardSetting": 2
}
SerialAllowAllPortsForUrls: ["http://localhost"]— pages served fromhttp://localhost(which is where the Octolux's Apache serves the HMI) get blanket permission to enumerate and open serial ports.getPorts()now returns the full list.DefaultSerialGuardSetting: 2—2means "Do not allow any site to request access". Belt-and-suspenders: if your code accidentally callsrequestPort(), the picker is a no-op instead of presenting UI nobody can see.
Path: /etc/chromium/policies/managed/octolux-serial.json. Restart Chromium after install (systemctl restart octolux-chromium). After restart, you must also bounce the screencast (see hmi-deploy-reset-vnc).
Verify the policy is live via CDP eval:
navigator.serial.getPorts().then(p => p.length)
// Expected: a positive integer (typically 9 on Octolux 15.6") when an RP
// device is plugged in. Zero means the policy didn't load — re-check
// the file path and the chromium restart.
One-time udev rule
The Octolux Yocto distro ships /dev/ttyACM0 as -rw-r--r-- root:root. The kiosk Chromium can't open it. Fix with a rule that targets the RP-series VID:
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'EOF'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
EOF
udevadm control --reload-rules
For an already-plugged device, the rule doesn't retroactively re-permission. Either physically unplug+replug, or cycle the USB driver from a shell:
echo 4-1 > /sys/bus/usb/drivers/usb/unbind
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind
(4-1 is the typical Octolux internal hub port; substitute from lsusb -t if your topology differs.)
Note: the Yocto distro lacks the usb_id udev helper, so udevadm info /dev/ttyACM0 returns "Unknown device: Inappropriate ioctl". That's fine. Chromium reads getInfo().usbVendorId from the underlying USB descriptor, not from udev's metadata database. You only need the permission rule, not metadata population.
Standard deploy sequence
Every HMI edit goes through this. Don't skip the screencast bounce — see hmi-deploy-reset-vnc.
ID=acme-control
DEVICE_IP=10.0.10.98
# 1. Push the HTML
scp ~/ntx-embedded/customers/$ID/hmi/index.html \
root@$DEVICE_IP:/usr/share/apache2/default-site/htdocs/index.html
# 2. Reload the kiosk — restart chromium pulls the new HTML AND
# reloads any new chromium policy / udev rule deployed alongside.
ssh root@$DEVICE_IP 'systemctl restart octolux-chromium'
# 3. Wait for the kiosk to come back
sleep 10
# 4. Bounce the screencast — chromium restart killed its CDP target
# binding and the screencast won't auto-reattach. Without this,
# the dashboard VNC tile freezes on a pre-deploy frame.
# NOTE: busybox `ps aux` truncates cmdlines so `ps aux | grep
# screencast` returns NOTHING even when it's running. Walk /proc
# instead. And NEVER walk /proc/PID/net/tcp to find port owners —
# those views are namespace-wide and you will mass-kill systemd.
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
for d in /proc/[0-9]*; do
pid=${d##*/}
grep -q 'screencast' "$d/cmdline" 2>/dev/null && kill -9 "$pid" 2>/dev/null
done
sleep 2
cd /tmp && nohup node screencast.js --quality 80 --pipeline 4 > /tmp/screencast.log 2>&1 &
REMOTE
# 5. (optional) confirm the dashboard VNC is alive — successive snaps
# should differ unless the page is genuinely static.
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/a.jpg -w "%{size_download}\n"
sleep 2
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/b.jpg -w "%{size_download}\n"
If /tmp/screencast.js or /tmp/node_modules/ws/ ever go missing on the device (they live in tmpfs and can be wiped by a bad kill spree), SCP them back from the container:
scp ~/ntx-embedded/apps/octolux/ui/screencast/screencast.js \
root@$DEVICE_IP:/tmp/screencast.js
cd ~/ntx-embedded/apps/octolux/node_modules && tar cz ws | \
ssh root@$DEVICE_IP 'mkdir -p /tmp/node_modules && cd /tmp/node_modules && tar xz'
Live diagnostics — CDP from the container
The Octolux kiosk exposes Chrome DevTools at localhost:9222. Tunnel it back:
ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP
# install python websocket client once
pip install --user --break-system-packages websocket-client
Then any time you want to inspect the live page state:
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def cdp(method, params=None, _id=[0]):
_id[0] += 1
ws.send(json.dumps({"id": _id[0], "method": method, "params": params or {}}))
while True:
r = json.loads(ws.recv())
if r.get("id") == _id[0]: return r
def evx(expr, await_=False):
return cdp("Runtime.evaluate", {"expression": expr, "returnByValue": True, "awaitPromise": await_})["result"]["result"]["value"]
# Useful queries:
evx("state.connected")
evx("state.fwVersion")
evx("navigator.serial.getPorts().then(p => p.length)", await_=True)
evx("Array.from(document.querySelectorAll('#debugStrip > div')).map(d=>d.textContent).join('\\n')")
# Force a reload (and remember: bounce screencast afterwards!):
cdp("Page.reload", {"ignoreCache": True})
Hit-test if a tap doesn't fire the button
const b = document.getElementById('btnConnect');
const r = b.getBoundingClientRect();
const hit = document.elementFromPoint(r.left + r.width/2, r.top + r.height/2);
JSON.stringify({ same: hit === b, hit_id: hit?.id, hit_tag: hit?.tagName });
If same is false, something is covering the button.
Failure-mode reference
Every row in this table corresponds to a real failure observed during this pattern's development. Use it as the diagnostic flow.
| Symptom | Root cause | Fix |
|---|---|---|
| Picker shows "No compatible devices found" | Chromium policy not in place AND/OR /dev/ttyACM* not world-readable |
Install both the policy and the udev rule; restart chromium; bounce screencast |
| Picker shows the device but every connect ends in NotFoundError: No port selected by user | User dismissed the picker, or requestPort() was called from a non-user-gesture context (programmatic CDP .click() triggers this) |
Don't use requestPort() at all — use the getPorts() auto-connect pattern in the HMI template |
getPorts() returns ports but port.open() throws Failed to execute 'open' on 'SerialPort': Failed to open serial port |
You opened ports[0]; it's a platform UART without a backing device |
Filter by getInfo().usbVendorId === 0x2e8a before opening |
getPorts().length === 0 even though lsusb shows the device on the kernel side |
Policy file missing, malformed, or wrong path; OR Chromium wasn't restarted after the policy install; OR /dev/ttyACM0 perms are still root-only |
Re-check /etc/chromium/policies/managed/octolux-serial.json is valid JSON; restart chromium; ls -la /dev/ttyACM0 should be 0666 |
| Tap on a button does nothing — no visible page reaction | Touch input might not be reaching the page; check via the debug strip (it should log pointerdown / click). If those don't appear → input layer (Weston/touchscreen) issue. If they do appear but connect() called doesn't → the click hit something covering the button — use elementFromPoint |
Diagnose via debug strip + hit-test |
The debug strip shows connect() called → requestPort()... and hangs — then eventually fails "No port selected by user" |
The HMI still has requestPort() somewhere |
Replace with getPorts() + VID filter (HMI template) |
Page reloaded via CDP Page.reload — VNC dashboard tile froze on the pre-reload frame |
The screencast bound to the old CDP target ID; reload creates a new one; the binding is dead | Bounce screencast (Standard deploy sequence step 4). Applies to ANY Chromium target change — restart, reload, navigate |
octolux deploy push times out |
Octolux server's deploy API is flaky on this build | Use the SCP + restart bash sequence directly (it's what the API does internally anyway) |
ps aux | grep screencast returns nothing but screencast IS running and bound to port 8085 |
Busybox ps aux truncates cmdlines |
Walk /proc/[0-9]*/cmdline to find PIDs by argv match |
| Killed a system PID and lost SSH / had to recover the device | You walked /proc/[0-9]*/net/tcp to find owners of port 8085 — those views are namespace-wide and match every PID |
NEVER identify port owners via /proc/PID/net/tcp on this distro. Use /proc/PID/cmdline matching instead |
/tmp/screencast.js or /tmp/node_modules/ws/ missing after a bad kill / reboot |
They live in tmpfs and the original placement path doesn't auto-restore | SCP from ~/ntx-embedded/apps/octolux/ui/screencast/screencast.js and tar+SCP ~/ntx-embedded/apps/octolux/node_modules/ws/ |
Adom HTTPS proxy returns error code: 502 with no body when the HMI's own dashboard endpoint fails |
The Adom proxy strips bodies on 5xx (Cloudflare-style stock error page) | Have the HMI server return HTTP 200 with {ok:false, error, hint} instead of 5xx — see octolux-app-debug Symptom 6 |
| Dashboard section button (VNC / Deploy / Stream) does nothing | Different issue — Hydrogen browser session disconnected or server.js handler — see octolux-app-debug Symptoms 5 and 6 |
Not in this skill |
Things NOT to do
- Don't blindly
open(ports[0])— always filter by VID. The kiosk policy returns the full set of/dev/tty*and most of them are platform UARTs that won't open. - Don't rely on
requestPort()— it's user-gesture-gated and shows a picker, and the picker is invisible to VNC, and the operator shouldn't have to tap anyway. - Don't walk
/proc/[0-9]*/net/tcpto find what's bound to a port. Namespace-wide view →xargs killhits systemd PIDs. - Don't
pkill nodeon the kiosk. node-red, the screencast, and other helpers all run asnode. Same rule that applies on the Adom container. - Don't restart Chromium without resetting screencast — see
hmi-deploy-reset-vnc. Same applies to CDPPage.reloadandPage.navigate. - Don't return 5xx for app errors that need an explanatory body — the Adom HTTPS proxy strips bodies on 5xx. Return HTTP 200 with
{ok: false, error, hint}. - Don't assume udev metadata is populated on the Yocto distro —
udevadm info /dev/ttyACM0returns "Inappropriate ioctl". Doesn't matter; Chromium reads VID/PID from the USB descriptor viagetInfo().
Related skills
hmi-deploy-reset-vnc(auto-memory) — the screencast-bounce rule. Triggered by ANY Chromium target change.octolux-hmi-design— visual design language (dark mode, accent colors, touch targets, Inter font) for the HTML body and CSS that wraps this skill's connect logic.octolux-platform— Octolux device-side reference (Weston, Chromium kiosk launcher, Apache config, paths).octolux-deploy— the wider HMI-deploy story; this skill's deploy sequence is a more reliable shell-equivalent ofoctolux deploy push.octolux-app-debug— when the issue is the Octolux Node server / dashboard, not the HMI/firmware pair.rp2350-flash— the SWD/flash side of the firmware lifecycle. This skill is the USB-runtime side; that one is the flash side.
# web-serial-hmi — NTX HMI ↔ USB CDC firmware pattern
> **Scope.** Building a Chromium-kiosk HMI on an **NTX Octolux device** that auto-connects to a **USB-attached RP2040 or RP2350** target over CDC-ACM, with **no Web Serial picker** ever showing. End-to-end working pattern, not a per-customer recipe — instantiate it for any new project under `~/ntx-embedded/customers/<id>/`.
>
> Out of scope: Android USB Host (different stack); Web Bluetooth / WebHID; non-RP-series USB devices (mostly the same pattern, just substitute the VID); CAN-FD bridges; the SWD/flash side of the firmware lifecycle — that's covered by `rp2350-flash` and the `target/rp2040.cfg` openocd patterns and the local `firmware/` build is just plain `cmake + make`.
## When to use this skill
- Starting a new customer HMI project that needs to control hardware over a USB-attached MCU.
- Porting an Android-tablet HMI app to an Octolux Linux/Chromium device and you want the same "just plug it in" feel.
- The Web Serial picker is showing — *"No compatible devices found"*, *"No port selected by user"*, or just appearing at all — and the project shouldn't have a picker.
- `requestPort()` works once but the operator has to tap-and-pick every session and that's wrong for a kiosk.
- The HMI page can't see what's happening because Chromium permission dialogs don't render in VNC.
## Architecture
```
┌─────────────────────────────────┐ USB-C ┌──────────────────────┐
│ NTX Octolux device (10.0.10.x) │ ◄──────────────► │ RP2040 / RP2350 board│
│ ─────────────────────────────── │ CDC-ACM, 115200 │ ─────────────────── │
│ Chromium 109 kiosk @ localhost │ 8N1 line ASCII │ Pico SDK USB CDC │
│ ├─ Web Serial API │ │ ├─ getchar polling │
│ ├─ SerialAllowAllPortsForUrls │ │ ├─ ASCII line parser│
│ └─ getPorts() filtered by VID │ │ └─ GPIO drive │
│ Apache at / │ │ │
│ /dev/ttyACM0 (root:root, 0666) │ │ │
└─────────────────────────────────┘ └──────────────────────┘
▲ ▲
│ SSH from container (deploy + udev + policy) │ SWD via PiProbe (flash)
│ │
Adom Docker container see rp2350-flash
~/ntx-embedded/customers/<id>/{hmi,firmware}
```
Two physical links from the workcell to the target board:
- **USB-C** between Octolux and target — runtime command path (this skill).
- **SWD** between the workcell's PiProbe and the target — flash path (`rp2350-flash`).
The container is the build host for both: HMI HTML edits go SCP-ed to Octolux; firmware ELF is built locally and flashed via the PiProbe.
## Required components
A working NTX HMI+firmware pair has **five** moving parts. Skip any one and connect won't happen. Pieces 3 and 4 are one-time setup *per Octolux device*; pieces 1, 2, 5 are per-project.
| # | Layer | Where it lives | Per-… |
|---|---|---|---|
| 1 | **Firmware** — USB CDC stdio, command parser, GPIO drive | `customers/<id>/firmware/src/main.c` (in container) | project |
| 2 | **HMI HTML** — connect logic + UI | `customers/<id>/hmi/index.html` (in container, deployed to device) | project |
| 3 | **Chromium policy** — `SerialAllowAllPortsForUrls` | `/etc/chromium/policies/managed/octolux-serial.json` (on device) | device |
| 4 | **udev rule** — `MODE="0666"` for VID 0x2e8a | `/etc/udev/rules.d/99-rp-cdc-acm.rules` (on device) | device |
| 5 | **Deploy** — SCP HMI + chromium restart + screencast bounce | shell pattern run from container | project |
## Quick start — new project, end to end
Run from the Adom container. Substitute `<id>` (e.g. `acme-control`), `<device-ip>` (e.g. `10.0.10.98`), and your domain commands.
### 1. Scaffold
```bash
ID=acme-control
DEVICE_IP=10.0.10.98
DEVICE_MODEL='Octolux 15.6"'
VIEW_W=1920 VIEW_H=1080
mkdir -p ~/ntx-embedded/customers/$ID/{hmi,firmware/src}
cat > ~/ntx-embedded/customers/$ID/project.json <<EOF
{
"id": "$ID",
"name": "Acme Control",
"customer": "Acme",
"summary": "One-line description of what this HMI does.",
"accent": "#e87a1e",
"accentDeep": "#a64f10",
"deviceModel": "$DEVICE_MODEL",
"viewport": { "w": $VIEW_W, "h": $VIEW_H },
"orientation": "landscape",
"defaultIP": "$DEVICE_IP",
"tagline": "one-line operator-facing tagline"
}
EOF
```
### 2. Drop in the firmware + HMI templates
Use the templates further down this doc verbatim, customizing only:
- **Firmware:** `FW_VERSION_STRING` (the `<NAME> FW x.y.z` boot banner), the `dispatch()` command list, and your specific GPIO pin numbers / domain logic.
- **HMI:** the visible controls (buttons, sliders, etc.), the brand colors from your `project.json`, the response-line handlers in `handleResponse()`.
Keep the Web Serial connect logic and the visible debug strip **unchanged** — that's the point of this skill.
### 3. Build + flash the firmware (first time)
```bash
export PICO_SDK_PATH=$HOME/pico-sdk
cd ~/ntx-embedded/customers/$ID/firmware
mkdir -p build && cd build && cmake .. && make -j$(nproc)
# Flash via PiProbe over SWD — pick the cfg for your chip:
openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg \
-c "adapter speed 5000; program ${ID//-/_}.elf verify reset exit"
# (or target/rp2350.cfg for an RP2350 — see rp2350-flash)
```
### 4. One-time device setup (Chromium policy + udev rule)
Skip if this Octolux device has been set up for Web Serial before. Verify by SSH'ing in: if `/etc/chromium/policies/managed/octolux-serial.json` and `/etc/udev/rules.d/99-rp-cdc-acm.rules` both exist, you're done.
```bash
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
set -e
# Chromium policy: SerialAllowAllPortsForUrls turns getPorts() into a
# blanket enumeration for http://localhost; DefaultSerialGuardSetting=2
# also disables requestPort() pickers as a safety net.
mkdir -p /etc/chromium/policies/managed
cat > /etc/chromium/policies/managed/octolux-serial.json <<'POL'
{
"SerialAllowAllPortsForUrls": ["http://localhost"],
"DefaultSerialGuardSetting": 2
}
POL
# udev: /dev/ttyACM* is root-only by default on the Yocto distro, and
# the kiosk Chromium can't open it. MODE=0666 fixes that for any
# Raspberry Pi (RP2040/RP2350) USB CDC device.
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'UDEV'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
UDEV
udevadm control --reload-rules
# Re-process any ALREADY-plugged-in device so the new rule applies now.
# (Without this, the perms only apply to *future* plug events.)
BUSPORT=$(lsusb -t 2>/dev/null | awk '/Pico|2e8a/ {sub(/^[ \t]+/, ""); print}' | head -1)
echo "USB device tree row: ${BUSPORT:-(none seen)}"
# Cycle the parent USB device (substitute bus-port from `lsusb -t` if your
# topology differs; 4-1 is the common Octolux internal hub port).
echo 4-1 > /sys/bus/usb/drivers/usb/unbind 2>/dev/null || true
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind 2>/dev/null || true
systemctl restart octolux-chromium
REMOTE
# Per the [[hmi-deploy-reset-vnc]] rule: chromium restart leaves the
# screencast bound to a dead CDP target. Bounce the screencast on the
# device — see the Deploy sequence below for the exact commands.
```
### 5. Deploy the HMI to the device
See the "Standard deploy sequence" section below — it's the same shell pattern you'll run on every HMI edit.
### 6. Verify it auto-connected (CDP eval from container)
```bash
ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP
python3 - <<'PY'
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def evx(expr, await_=False):
ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
"params": {"expression": expr, "returnByValue": True,
"awaitPromise": await_}}))
while True:
r = json.loads(ws.recv())
if r.get("id") == 1: return r["result"]["result"]["value"]
print("connected:", evx("state.connected"))
print("fwVersion:", evx("state.fwVersion"))
print("ports :", evx("navigator.serial.getPorts().then(p => p.length)", await_=True))
PY
```
Expected once the firmware is running and the cable is plugged in:
```
connected: True
fwVersion: 1.0.0
ports : 9 # 8 platform UARTs + your 1 RP-series CDC
```
If `connected: True` and `fwVersion` is populated → you're done; the HMI is talking to the MCU. If not, jump to the failure table.
## Firmware template (RP2040 / RP2350 USB CDC)
`firmware/CMakeLists.txt`:
```cmake
cmake_minimum_required(VERSION 3.13)
set(PICO_BOARD pico CACHE STRING "") # or "pico2" for RP2350
set(PICO_PLATFORM rp2040 CACHE STRING "") # or "rp2350"
include($ENV{PICO_SDK_PATH}/external/pico_sdk_import.cmake)
project(<id_underscored> C CXX ASM)
pico_sdk_init()
add_executable(<id_underscored> src/main.c)
target_link_libraries(<id_underscored> pico_stdlib hardware_gpio)
pico_enable_stdio_usb(<id_underscored> 1) # CDC-ACM on USB-C
pico_enable_stdio_uart(<id_underscored> 0) # no UART — USB only
pico_add_extra_outputs(<id_underscored>)
```
`firmware/src/main.c`:
```c
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdarg.h>
#include "pico/stdlib.h"
#include "pico/stdio_usb.h" // stdio_usb_connected()
#include "hardware/gpio.h"
#define FW_VERSION_STRING "<NAME> FW 1.0.0"
#define CMD_BUF_SIZE 96
// Status LEDs (wire your board's status LEDs to these or change as needed).
// LED_USB: solid ON while USB CDC is connected to a host.
// LED_ACT: 50 ms blink on every valid command.
// LED_ERR: solid ON when the last response was ERR; clears on next OK.
// LED_IDLE: solid ON when system is in its safe / inactive state.
#define LED_USB 9
#define LED_ACT 10
#define LED_ERR 16
#define LED_IDLE 18
static char cmd_buf[CMD_BUF_SIZE];
static int cmd_len = 0;
static bool act_on = false;
static absolute_time_t act_off_at;
static void blink_activity(void) {
gpio_put(LED_ACT, 1);
act_on = true;
act_off_at = make_timeout_time_ms(50);
}
static void say_ok(const char *fmt, ...) {
va_list a; va_start(a, fmt);
fputs("OK ", stdout); vprintf(fmt, a); fputs("\r\n", stdout);
va_end(a);
gpio_put(LED_ERR, 0);
blink_activity();
}
static void say_err(const char *m) {
printf("ERR %s\r\n", m);
gpio_put(LED_ERR, 1);
blink_activity();
}
static void to_upper(char *s) { for (; *s; s++) *s = (char)toupper((unsigned char)*s); }
// === Domain commands ====================================================
//
// Replace these with the commands your project actually needs. Keep
// the canonical built-ins (PING, VERSION, STATUS, HELP) — the HMI
// template's handshake assumes they exist.
//
static void dispatch(char *line) {
char *tok = strtok(line, " \t");
if (!tok) return;
if (!strcmp(tok, "PING")) { say_ok("PONG"); return; }
if (!strcmp(tok, "VERSION")) { printf("%s\r\n", FW_VERSION_STRING); blink_activity(); return; }
if (!strcmp(tok, "STATUS")) { /* print one-line state snapshot */ blink_activity(); return; }
if (!strcmp(tok, "HELP")) { /* print "OK …" lines */ blink_activity(); return; }
// your-domain command parsing here, e.g.:
// if (!strcmp(tok, "DO")) { … say_ok("DO %d", arg); return; }
say_err("unknown command");
}
// === Boilerplate main =================================================
int main(void) {
stdio_init_all();
// Init your GPIOs here (LEDs + outputs). Drive outputs LOW first
// so the system boots in a safe state regardless of host behavior.
const uint leds[] = { LED_USB, LED_ACT, LED_ERR, LED_IDLE };
for (size_t i = 0; i < sizeof(leds)/sizeof(leds[0]); i++) {
gpio_init(leds[i]); gpio_set_dir(leds[i], GPIO_OUT); gpio_put(leds[i], 0);
}
gpio_put(LED_IDLE, 1);
bool was_conn = false, banner_done = false;
while (true) {
bool conn = stdio_usb_connected();
gpio_put(LED_USB, conn ? 1 : 0);
if (conn && !was_conn) banner_done = false;
was_conn = conn;
// Boot banner on every CDC connect — the HMI parses this to
// populate the FW version pill.
if (conn && !banner_done) {
sleep_ms(50); // give the host's reader a beat
printf("%s\r\n", FW_VERSION_STRING);
puts("TYPE 'HELP' FOR COMMANDS");
banner_done = true;
}
// Drain available input into cmd_buf; dispatch on EOL.
for (int budget = 0; budget < 32; budget++) {
int c = getchar_timeout_us(0);
if (c == PICO_ERROR_TIMEOUT) break;
if (c == '\n' || c == '\r') {
if (cmd_len > 0) {
cmd_buf[cmd_len] = '\0';
to_upper(cmd_buf);
dispatch(cmd_buf);
cmd_len = 0;
}
} else if (cmd_len < CMD_BUF_SIZE - 1) {
cmd_buf[cmd_len++] = (char)c;
} else { cmd_len = 0; say_err("command too long"); }
}
// Activity LED timeout.
if (act_on && absolute_time_diff_us(get_absolute_time(), act_off_at) <= 0) {
gpio_put(LED_ACT, 0); act_on = false;
}
}
}
```
### Protocol conventions that work well
- **Newline-terminated ASCII** — accept `\n` and `\r\n`. Case-insensitive (uppercase before dispatch).
- **One line in, exactly one line out** — `OK …`, `ERR …`, or a single-token status line. One-to-one keeps the HMI's send/receive trivial.
- **Standard built-ins always present:**
- `PING → PONG` (liveness)
- `VERSION → <NAME> FW x.y.z` (capability detection + version pill)
- `STATUS → STATE …` (one-line state snapshot)
- `HELP → multi-line OK …` (operator self-discovery)
- **Boot banner on every CDC connect.** Print `<NAME> FW x.y.z\r\n` first thing after `stdio_usb_connected()` flips true. The HMI matches `/^([\w-]+) FW (\S+)/i` to populate the version pill.
- **LED1 (USB) mirrors `stdio_usb_connected()`** — instant visual confirmation that the host opened the port.
- **LED2 (activity) blinks 50 ms on every valid command** — independent visual signal that the firmware *parsed* a command.
- **Default-safe on boot** — all outputs LOW, critical actuators OFF, idle LED ON. Don't rely on the host to send `ALL OFF` first.
## HMI template
```html
<script>
"use strict";
const state = {
port: null, writer: null, reader: null,
connected: false, fwVersion: null,
autoConnectTimer: null,
};
// ─── Visible debug strip ──────────────────────────────────────────────
// Mirror every UI event + Web Serial step into the page DOM, because
// Chromium permission dialogs (the Web Serial picker, geo prompts,
// etc.) render above the page surface and DON'T appear in the
// screencast / VNC feed. Without this strip, you'd be blind to what's
// happening on the kiosk from your dashboard.
function dbg(line) {
const strip = document.getElementById('debugStrip');
if (!strip) return console.log(line);
const d = document.createElement('div');
d.textContent = new Date().toISOString().slice(11,23) + ' ' + line;
strip.appendChild(d);
while (strip.children.length > 5) strip.removeChild(strip.firstChild);
}
// ─── Line-splitter transformer for the reader stream ──────────────────
function lineSplitter() {
let buf = '';
return {
transform(chunk, ctrl) {
buf += chunk;
let i;
while ((i = buf.search(/\r?\n/)) >= 0) {
ctrl.enqueue(buf.slice(0, i));
buf = buf.slice(i + (buf[i] === '\r' ? 2 : 1));
}
},
};
}
async function openExisting(port) {
await port.open({
baudRate: 115200, dataBits: 8, stopBits: 1, parity: 'none',
bufferSize: 256, flowControl: 'none',
});
state.port = port;
const enc = new TextEncoderStream();
enc.readable.pipeTo(port.writable);
state.writer = enc.writable.getWriter();
const dec = new TextDecoderStream();
port.readable.pipeTo(dec.writable);
state.reader = dec.readable
.pipeThrough(new TransformStream(lineSplitter()))
.getReader();
state.connected = true;
readLoop();
handshake();
}
async function tryAutoConnect() {
if (state.connected || !('serial' in navigator)) return state.connected;
const ports = await navigator.serial.getPorts();
// The kiosk policy returns the FULL set of /dev/tty* devices the
// kernel sees — typically ~8 platform UARTs (empty getInfo()) plus
// your real USB CDC. Always filter; never open ports[0] blindly.
// 0x2e8a is the Raspberry Pi VID (RP2040 + RP2350). Substitute your
// chip's VID if you use a non-RP target.
const rp = ports.filter(p => (p.getInfo() || {}).usbVendorId === 0x2e8a);
const anyUsb = ports.filter(p => typeof (p.getInfo() || {}).usbVendorId === 'number');
const candidates = rp.length ? rp : anyUsb;
dbg(`getPorts() → ${ports.length} total, ${rp.length} RP-series`);
for (const port of candidates) {
try {
await openExisting(port);
const info = port.getInfo();
dbg(`auto-connected to VID:${(info.usbVendorId||0).toString(16)} PID:${(info.usbProductId||0).toString(16)}`);
return true;
} catch (e) {
dbg(' open failed on one candidate: ' + e.message);
}
}
return false;
}
window.addEventListener('load', async () => {
// Install global capture-phase event listeners so we can see every
// tap on the page, even ones that don't land on a button.
['pointerdown','pointerup','touchstart','touchend',
'mousedown','mouseup','click'].forEach(k => {
document.addEventListener(k, e => {
const tgt = e.target ? (e.target.id || e.target.tagName) : '?';
dbg(`${k} (${(e.clientX||0)|0},${(e.clientY||0)|0}) → ${tgt} trusted=${e.isTrusted}`);
}, /* useCapture */ true);
});
dbg('debug listeners installed');
if (!(await tryAutoConnect())) {
// Keep polling — if the cable is plugged in *after* the page loads,
// the next tick picks it up. Cheap (one promise every 2 s).
state.autoConnectTimer = setInterval(async () => {
if (await tryAutoConnect() && state.autoConnectTimer) {
clearInterval(state.autoConnectTimer);
state.autoConnectTimer = null;
}
}, 2000);
}
});
async function readLoop() {
while (state.reader) {
try {
const { value, done } = await state.reader.read();
if (done) { state.connected = false; break; }
if (value) handleResponse(value.trim());
} catch { state.connected = false; break; }
}
}
async function handshake() {
await send('PING');
await send('VERSION');
await send('STATUS');
}
async function send(cmd) {
if (!state.writer) return;
dbg('→ ' + cmd);
await state.writer.write(cmd + '\n');
}
function handleResponse(line) {
dbg('← ' + line);
const m = line.match(/^([\w-]+) FW (\S+)/i);
if (m) state.fwVersion = m[2];
// Your domain handlers here (parse STATUS reply, react to OK/ERR, etc.)
}
</script>
```
The HMI's HTML body and CSS are entirely up to your project — see `octolux-hmi-design` for the brand-language guide (dark mode, accent colors, touch-target sizes, Inter font). The only DOM element this template requires is `<div id="debugStrip"></div>` somewhere visible during development. You can hide it (or remove the global tap listeners) after the project ships.
## One-time Chromium policy — the single biggest enabler
> Without `SerialAllowAllPortsForUrls`, `requestPort()` shows a picker, the picker isn't visible through VNC, and the kiosk is unreliable. With it, `getPorts()` returns every kernel serial device with no user gesture.
```json
{
"SerialAllowAllPortsForUrls": ["http://localhost"],
"DefaultSerialGuardSetting": 2
}
```
- **`SerialAllowAllPortsForUrls: ["http://localhost"]`** — pages served from `http://localhost` (which is where the Octolux's Apache serves the HMI) get blanket permission to enumerate and open serial ports. `getPorts()` now returns the full list.
- **`DefaultSerialGuardSetting: 2`** — `2` means *"Do not allow any site to request access"*. Belt-and-suspenders: if your code accidentally calls `requestPort()`, the picker is a no-op instead of presenting UI nobody can see.
Path: `/etc/chromium/policies/managed/octolux-serial.json`. Restart Chromium after install (`systemctl restart octolux-chromium`). After restart, you **must** also bounce the screencast (see `hmi-deploy-reset-vnc`).
Verify the policy is live via CDP eval:
```javascript
navigator.serial.getPorts().then(p => p.length)
// Expected: a positive integer (typically 9 on Octolux 15.6") when an RP
// device is plugged in. Zero means the policy didn't load — re-check
// the file path and the chromium restart.
```
## One-time udev rule
The Octolux Yocto distro ships `/dev/ttyACM0` as `-rw-r--r-- root:root`. The kiosk Chromium can't open it. Fix with a rule that targets the RP-series VID:
```bash
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'EOF'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
EOF
udevadm control --reload-rules
```
For an **already-plugged** device, the rule doesn't retroactively re-permission. Either physically unplug+replug, or cycle the USB driver from a shell:
```bash
echo 4-1 > /sys/bus/usb/drivers/usb/unbind
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind
```
(`4-1` is the typical Octolux internal hub port; substitute from `lsusb -t` if your topology differs.)
Note: the Yocto distro lacks the `usb_id` udev helper, so `udevadm info /dev/ttyACM0` returns *"Unknown device: Inappropriate ioctl"*. **That's fine.** Chromium reads `getInfo().usbVendorId` from the underlying USB descriptor, not from udev's metadata database. You only need the permission rule, not metadata population.
## Standard deploy sequence
Every HMI edit goes through this. Don't skip the screencast bounce — see `hmi-deploy-reset-vnc`.
```bash
ID=acme-control
DEVICE_IP=10.0.10.98
# 1. Push the HTML
scp ~/ntx-embedded/customers/$ID/hmi/index.html \
root@$DEVICE_IP:/usr/share/apache2/default-site/htdocs/index.html
# 2. Reload the kiosk — restart chromium pulls the new HTML AND
# reloads any new chromium policy / udev rule deployed alongside.
ssh root@$DEVICE_IP 'systemctl restart octolux-chromium'
# 3. Wait for the kiosk to come back
sleep 10
# 4. Bounce the screencast — chromium restart killed its CDP target
# binding and the screencast won't auto-reattach. Without this,
# the dashboard VNC tile freezes on a pre-deploy frame.
# NOTE: busybox `ps aux` truncates cmdlines so `ps aux | grep
# screencast` returns NOTHING even when it's running. Walk /proc
# instead. And NEVER walk /proc/PID/net/tcp to find port owners —
# those views are namespace-wide and you will mass-kill systemd.
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
for d in /proc/[0-9]*; do
pid=${d##*/}
grep -q 'screencast' "$d/cmdline" 2>/dev/null && kill -9 "$pid" 2>/dev/null
done
sleep 2
cd /tmp && nohup node screencast.js --quality 80 --pipeline 4 > /tmp/screencast.log 2>&1 &
REMOTE
# 5. (optional) confirm the dashboard VNC is alive — successive snaps
# should differ unless the page is genuinely static.
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/a.jpg -w "%{size_download}\n"
sleep 2
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/b.jpg -w "%{size_download}\n"
```
If `/tmp/screencast.js` or `/tmp/node_modules/ws/` ever go missing on the device (they live in tmpfs and can be wiped by a bad kill spree), SCP them back from the container:
```bash
scp ~/ntx-embedded/apps/octolux/ui/screencast/screencast.js \
root@$DEVICE_IP:/tmp/screencast.js
cd ~/ntx-embedded/apps/octolux/node_modules && tar cz ws | \
ssh root@$DEVICE_IP 'mkdir -p /tmp/node_modules && cd /tmp/node_modules && tar xz'
```
## Live diagnostics — CDP from the container
The Octolux kiosk exposes Chrome DevTools at `localhost:9222`. Tunnel it back:
```bash
ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP
# install python websocket client once
pip install --user --break-system-packages websocket-client
```
Then any time you want to inspect the live page state:
```python
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def cdp(method, params=None, _id=[0]):
_id[0] += 1
ws.send(json.dumps({"id": _id[0], "method": method, "params": params or {}}))
while True:
r = json.loads(ws.recv())
if r.get("id") == _id[0]: return r
def evx(expr, await_=False):
return cdp("Runtime.evaluate", {"expression": expr, "returnByValue": True, "awaitPromise": await_})["result"]["result"]["value"]
# Useful queries:
evx("state.connected")
evx("state.fwVersion")
evx("navigator.serial.getPorts().then(p => p.length)", await_=True)
evx("Array.from(document.querySelectorAll('#debugStrip > div')).map(d=>d.textContent).join('\\n')")
# Force a reload (and remember: bounce screencast afterwards!):
cdp("Page.reload", {"ignoreCache": True})
```
### Hit-test if a tap doesn't fire the button
```javascript
const b = document.getElementById('btnConnect');
const r = b.getBoundingClientRect();
const hit = document.elementFromPoint(r.left + r.width/2, r.top + r.height/2);
JSON.stringify({ same: hit === b, hit_id: hit?.id, hit_tag: hit?.tagName });
```
If `same` is false, something is covering the button.
## Failure-mode reference
Every row in this table corresponds to a real failure observed during this pattern's development. Use it as the diagnostic flow.
| Symptom | Root cause | Fix |
|---|---|---|
| Picker shows *"No compatible devices found"* | Chromium policy not in place AND/OR `/dev/ttyACM*` not world-readable | Install both the policy and the udev rule; restart chromium; bounce screencast |
| Picker shows the device but every connect ends in *NotFoundError: No port selected by user* | User dismissed the picker, or `requestPort()` was called from a non-user-gesture context (programmatic CDP `.click()` triggers this) | Don't use `requestPort()` at all — use the `getPorts()` auto-connect pattern in the HMI template |
| `getPorts()` returns ports but `port.open()` throws *Failed to execute 'open' on 'SerialPort': Failed to open serial port* | You opened `ports[0]`; it's a platform UART without a backing device | Filter by `getInfo().usbVendorId === 0x2e8a` before opening |
| `getPorts().length === 0` even though `lsusb` shows the device on the kernel side | Policy file missing, malformed, or wrong path; OR Chromium wasn't restarted after the policy install; OR `/dev/ttyACM0` perms are still root-only | Re-check `/etc/chromium/policies/managed/octolux-serial.json` is valid JSON; restart chromium; `ls -la /dev/ttyACM0` should be `0666` |
| Tap on a button does nothing — no visible page reaction | Touch input might not be reaching the page; check via the debug strip (it should log `pointerdown` / `click`). If those don't appear → input layer (Weston/touchscreen) issue. If they do appear but `connect() called` doesn't → the click hit something covering the button — use `elementFromPoint` | Diagnose via debug strip + hit-test |
| The debug strip shows `connect() called → requestPort()...` and hangs — then eventually fails *"No port selected by user"* | The HMI still has `requestPort()` somewhere | Replace with `getPorts()` + VID filter (HMI template) |
| Page reloaded via CDP `Page.reload` — VNC dashboard tile froze on the pre-reload frame | The screencast bound to the old CDP target ID; reload creates a new one; the binding is dead | Bounce screencast (Standard deploy sequence step 4). Applies to ANY Chromium target change — restart, reload, navigate |
| `octolux deploy push` times out | Octolux server's deploy API is flaky on this build | Use the SCP + restart bash sequence directly (it's what the API does internally anyway) |
| `ps aux \| grep screencast` returns nothing but screencast IS running and bound to port 8085 | Busybox `ps aux` truncates cmdlines | Walk `/proc/[0-9]*/cmdline` to find PIDs by argv match |
| Killed a system PID and lost SSH / had to recover the device | You walked `/proc/[0-9]*/net/tcp` to find owners of port 8085 — those views are namespace-wide and match every PID | **NEVER** identify port owners via `/proc/PID/net/tcp` on this distro. Use `/proc/PID/cmdline` matching instead |
| `/tmp/screencast.js` or `/tmp/node_modules/ws/` missing after a bad kill / reboot | They live in tmpfs and the original placement path doesn't auto-restore | SCP from `~/ntx-embedded/apps/octolux/ui/screencast/screencast.js` and tar+SCP `~/ntx-embedded/apps/octolux/node_modules/ws/` |
| Adom HTTPS proxy returns `error code: 502` with no body when the HMI's own dashboard endpoint fails | The Adom proxy strips bodies on 5xx (Cloudflare-style stock error page) | Have the HMI server return HTTP 200 with `{ok:false, error, hint}` instead of 5xx — see `octolux-app-debug` Symptom 6 |
| Dashboard section button (VNC / Deploy / Stream) does nothing | Different issue — Hydrogen browser session disconnected or server.js handler — see `octolux-app-debug` Symptoms 5 and 6 | Not in this skill |
## Things NOT to do
- **Don't blindly `open(ports[0])`** — always filter by VID. The kiosk policy returns the full set of `/dev/tty*` and most of them are platform UARTs that won't open.
- **Don't rely on `requestPort()`** — it's user-gesture-gated and shows a picker, and the picker is invisible to VNC, and the operator shouldn't have to tap anyway.
- **Don't walk `/proc/[0-9]*/net/tcp`** to find what's bound to a port. Namespace-wide view → `xargs kill` hits systemd PIDs.
- **Don't `pkill node`** on the kiosk. node-red, the screencast, and other helpers all run as `node`. Same rule that applies on the Adom container.
- **Don't restart Chromium without resetting screencast** — see `hmi-deploy-reset-vnc`. Same applies to CDP `Page.reload` and `Page.navigate`.
- **Don't return 5xx** for app errors that need an explanatory body — the Adom HTTPS proxy strips bodies on 5xx. Return HTTP 200 with `{ok: false, error, hint}`.
- **Don't assume udev metadata is populated** on the Yocto distro — `udevadm info /dev/ttyACM0` returns *"Inappropriate ioctl"*. Doesn't matter; Chromium reads VID/PID from the USB descriptor via `getInfo()`.
## Related skills
- `hmi-deploy-reset-vnc` (auto-memory) — the screencast-bounce rule. Triggered by ANY Chromium target change.
- `octolux-hmi-design` — visual design language (dark mode, accent colors, touch targets, Inter font) for the HTML body and CSS that wraps this skill's connect logic.
- `octolux-platform` — Octolux device-side reference (Weston, Chromium kiosk launcher, Apache config, paths).
- `octolux-deploy` — the wider HMI-deploy story; this skill's deploy sequence is a more reliable shell-equivalent of `octolux deploy push`.
- `octolux-app-debug` — when the issue is the Octolux Node server / dashboard, not the HMI/firmware pair.
- `rp2350-flash` — the SWD/flash side of the firmware lifecycle. This skill is the USB-runtime side; that one is the flash side.