Closed feature request

Make wiki pages drivable by the AI: a JS API (or live channel) for viewers, widgets and scrolling, instead of mouse clicks

John Lauer · 11d ago ·edited ·closed by John Lauer

@kcknox, this one is about making every Adom surface drivable by the AI, starting with the wiki, and it will hit your Altium Bridge video the same way it is hitting my Fusion Bridge video.

What happened

For the Fusion buck demo video, the AI had to show four component pages: open the page, load the 3D model, then on the LED page jump to States, load the states widget and flip it from Off to On. There is no way for the AI to ask the page to do any of that, so it drove it like a person would:

  • turned on Adom Bridge mouse control, then clicked screen pixels for "Load all viewers", "Load 3D model", "Load adom-states" and the On button;
  • guessed coordinates from screenshots. One "Load 3D model" click landed 23 px above the button and the TPS54202 model never loaded; it took three re-takes to find out why;
  • "Load all viewers" is a toggle, so on a page that already had viewers loaded, the same click unloaded them;
  • had no way to ask the page "is the model loaded yet?", so every step waited a fixed number of seconds and hoped.

It works, but it is slow, brittle and it looks clunky on camera. Every time the AI has to fall back to mouse driving, that is a surface the AI cannot actually control.

Why it matters now

Your Altium Bridge video is the same shape as this Fusion one: the AI pulls parts from the wiki, shows their pages and 3D models, then builds the board. You will hit exactly this. It is also the general rule we want: every Adom surface should be drivable by the AI directly, not through the mouse.

Two ways to do it

1. A small JavaScript API on every wiki page (lighter, and probably the place to start). Expose something like window.adomWiki and let the AI drive it with an eval, through whatever is showing the page: a Hydrogen webview, a pup window or the native browser. All three can already run JavaScript in a page. For example:

await adomWiki.viewers.loadAll()                 // the same as the button, but not a toggle
await adomWiki.viewer('3d').ready()              // resolves when the model has rendered
adomWiki.viewer('3d').orbit({ yaw: 30, seconds: 2 })
adomWiki.viewer('3d').setCamera('iso')
adomWiki.scrollTo('states', { smooth: true })
await adomWiki.widget('states').load()
adomWiki.widget('states').set('on')              // the LED lights up
adomWiki.state()                                 // which viewers/widgets are loaded, current camera, scroll

Each call returns a promise that resolves when the thing is actually on screen, so the AI never sleeps and guesses. This keeps #1292 intact: viewers still never load on their own, the AI loads them on purpose.

2. A live channel back to the AI (heavier, more powerful). A URL parameter that names an SSE endpoint, for example ?drive=<endpoint>. The page connects back to it, and a server-side process in the user's container can then send commands and receive state and events, the same pattern the app-creator skill's live channel uses for our own apps. The AI talks to that process instead of injecting JavaScript, and gets pushed events ("viewer ready", "user clicked On") instead of polling.

Option 1 is enough for demo videos and most AI flows. Option 2 is worth it if we want the AI and a person to share a wiki page live.

Ask

  1. Pick a direction (I would start with option 1).
  2. Make the component page viewers and widgets controllable through it first: 3D viewer load, ready, orbit and camera; the states widget; scroll to a section; the inline datasheet.
  3. Document it in the wiki skill so every agent finds it, and have the 3D viewer and widget code expose the same hooks so new widgets are drivable by default.
  4. Then we retire the mouse-driven fallback for wiki pages in both demo videos.

5 Replies

John Lauer · 11d ago

@kcknox tagging you directly, since the mention in the body was fixed after posting. This will matter for the Altium Bridge video too.

John Lauer · 11d ago

Hydrogen Desktop now has the webview postMessage action: 1.0.584, on the insiders tier. It follows the handoff contract, so adom-cli hydrogen webview drive (0.7.14) works against HD.

What changed in HD:

  • The postMessage action: hydrogen-proxy handles PATCH /api/panels/webview/{owner}/{repo} {action:"postMessage"} and the new PUT on the same route. Timeouts are 200 with phase; no_frame 409, not_found 404, not_webview 400, bad requests 400, and 409 when no window is connected.
  • The frontend: webviewBridge.ts is copied unchanged. PanelWebView registers its iframe with whenLoaded and hasApplied, and PanelWorkspaceComponent resolves the tab selector and relays.
  • Step 0: HD now handles webview_control, so agent navigate, refresh, setHeaderHidden and setProxyEnabled actually work. Navigate also keeps the tab's name (HD used to rename the tab after the new page's title, so the next --name call got not_found).

Tested on Windows (ConfRoomROG, 1.0.584), https://wiki.adom.inc/adom/stm32f303cbt6 in a Web View named STM32, proxy mode off:

  • page: answers adom/stm32f303cbt6.
  • 3D ready: loaded: true.
  • 3D orbit {degrees:360, seconds:6}: answered after 6.08 s.
  • Datasheet page 3: page: 3, total: 148.
  • eval: 200 with the wiki's ok: false "unknown method".
  • An ignored message.type with timeoutMs: 2000: 200 status: timeout, phase: reply.
  • Navigate, then an immediate page: answered by the new page.
  • Unknown tab name: 404. External Webview tab: 400 not_webview.

Not done:

  • Proxy mode on: the page now loads its scripts and answers (eval and the timeouts behave), but HD's proxy serves every page at /webview-proxy. So page returns ref: null and the viewers never start. Filed as adom/hydrogen#108. Leave proxy mode off for driving.
  • macOS: not tested; no Mac was connected.
Colby Knox · 11d ago

Thanks for running it end to end on HD. That covers the whole lane: the wiki's message bridge, the HD relay, and adom-cli hydrogen webview drive.

Proxy mode, the wiki side. adomWiki.page() now reads the page's <link rel="canonical"> instead of location.pathname. A page served from /webview-proxy reports its real ref (adom/stm32f303cbt6), not null. The change is on staging now and goes to prod with the next tag.

Proxy mode, still HD's (adom/hydrogen#108). The viewers load their frames from relative URLs (/viewer/3d/..., /viewer/datasheet/...), so a proxy that serves the page at /webview-proxy needs to rewrite or route those paths too. Until #108 lands, keep proxy mode off for driving.

John Lauer · 10d ago

Confirmed from the Fusion video side: adom-cli hydrogen webview drive covers every step I had to click through for the component shots (viewers.loadAll, 3D ready and orbit, scrollTo, and set ["on"] --widget adom-states for the LED). Next time those shots get driven this way instead of the mouse. Closing this; the proxy-mode remainder is tracked in adom/hydrogen#108. Thanks @kcknox.

John Lauer · 10d ago

Update: proxy mode works too, in Hydrogen Desktop 1.0.585 (insiders). The whole test list now passes on Windows with proxy mode both off and on: page, 3D ready, orbit 360/6 s (6.11 s), datasheet page 3/148, eval, the ignored-type timeout, navigate-then-page, 404/400.

HD's proxy mode used to serve every page at /webview-proxy. It now gives each site its own loopback origin, <base32>.wvp.localhost, which keeps the real path. That also closed a hole: proxied pages used to run on the control API's own origin. Details are in adom/hydrogen#108 (closed).

macOS is still untested.

Log in to reply.