Closed bug report

AI Flow takes over the user's screen: every step raises KiCad or Hydrogen to the front

noah · ·closed by John Lauer

AI Flow takes over the user's screen: every step raises KiCad or Hydrogen to the front

Seen on: adom-aiflow 0.2.40 to 0.2.44, the AM32 ESC run (adom/am32-esc-2s6s-molecule) on the user's own Windows PC (Hydrogen local workspace, Adom Bridge on the same machine, KiCad 10.0.6).

What happened: throughout the run, KiCad (and Hydrogen) kept being brought to the foreground while the user was working on their own PC. The user had to fight the AI for their own computer. In the source this is by design:

  • step (main.rs, capture block) calls bring_front on Hydrogen at every new phase and on 11 "Hydrogen steps". It calls it on KiCad's PCB editor on every board step: desktop_bring_to_front with "show KiCad for this step".
  • capture open → Bridge::open_board (aiflow-bridge lib.rs) opens the board with foreground: true, maximises the PCB editor, then brings it to the front again.
  • tour 3d maximises the 3D viewer and brings it to the front.

The comments say "on the test box". On a Hydrogen local workspace, the test box is the person's own desktop, and the run has 25 steps plus every return.

The only off switch is the environment variable AIFLOW_AUTO_CAPTURE=off. The skill never mentions it, and it also turns off the step clips.

Why it is wrong: AI Flow exists so the AI can work on the board while the person does something else. Taking the screen defeats that, and it is unsafe: a keystroke or click meant for the person's own window can land in KiCad. The window recorder already used for clips (Windows Graphics Capture) is background-capturable, as the code itself notes, so filming does not need the foreground. The window only must not be minimised.

Suggested fix:

  1. Never take the foreground on the person's own machine unless they said so.
    • Default screen: "hands-off" whenever the target is the machine the human is using (Hydrogen local workspace, or ab reports an active interactive session), and keep the current behaviour only for a declared test box (--target <box> with screen: "allowed").
    • In hands-off mode, skip desktop_bring_to_front and maximize in step, capture open and tour 3d.
    • Open the board un-maximised, without focus.
    • Record by window capture. If a window is minimised or fully hidden when a clip needs it, record a skipped clip with the reason (as issue #59 already does for idle windows) and say so in the run README, rather than raising it.
  2. Make it a preference, recorded in run.json and shown on the timeline: screen: hands-off | ask | allowed. Let a person change it mid-run ("you can take my screen now" → allowed until they say otherwise).
  3. Questions to the human are the one exception, also by preference: notify: toast | toast+sound | hydrogen-only | none.
    • A toast (ab notify_user) or a Hydrogen notification, never a window raise.
    • The answer can come back through the toast's buttons or the chat.
  4. Document both in SKILL.md, next to "Capture, so the engines' videos line up", and in the foreground-etiquette skill.

A per-user preference system for this (and for how much the AI does on its own at each step) is being proposed in a separate PR, "hands-on AI Flow", which will link here.

2 Replies

noah mentioned this in pull request #59 ·
noah ·

The preference system mentioned above is PR #59 (hands-on AI Flow). Every flow profile carries screen: hands-off by default, which skips the per-step raises in step, opens the board without focus or maximise in capture open, and makes tour 3d ask the human first. notify (hydrogen, toast, both, none) is how the AI reaches them instead. Runs without a profile keep today's behaviour, so the default for everyone is still open here.

John Lauer ·

Noah, fixed in adom-aiflow 0.2.49. I merged it together with your PR #59, so there is now one screen rule.

The default comes from the machine. On the machine the human is using, the screen is hands-off: nothing is raised or maximised, the board opens without focus, and clips record by window capture. A declared test box keeps allowed. AI Flow says which mode it chose and why, once, in the run.

The human can change it at any time, and the latest word wins:

  • capture screen hands-off | ask | allowed mid-run;
  • your profile's handson set --screen;
  • capture.screen in the spec;
  • AIFLOW_SCREEN for a single command.

tour 3d on someone else's screen: the 3D viewer only paints when it's in front, so the tour is skipped there. The skip comes with a suggestion: ask the human, or have them run it. It no longer refuses, and finish doesn't need it.

The run records the mode in run.json screenMode. Each step logs screen-hands-off when it left a window where it was.

Upgrade: tell your AI "give me the latest adom-aiflow" (or adom-wiki pkg install adom/adom-aiflow); adom-aiflow --version says 0.2.49.

John Lauer closed this ·

Log in to reply.