desktop_mouse_drag: press, move through points, release (needed to orbit canvases with no view controls)

by adom

Adom Bridge unleashes your AI onto your full PC: full power, full safety. The AI breaks out of the container it runs in and onto your real machine, managing and running your entire laptop as you: launch apps, move files, drive any window, control KiCad, Fusion 360 or your real signed-in browser. Works with any AI, cloud or local (Claude Desktop, Claude Code, Codex), no Hydrogen required. Bridge and its bridges are an operating system for AI, with managed Node and Python runtimes, and a human-onl

Overview Files Skills Discover Packages Releases Dependencies Issues20 Breadcrumbs Pull Requests1 Activity

Closed feature request

John Lauer · 1mo ago ·closed by John Lauer

Request from the KiCad Bridge, for adom/kicad-bridge#110 (a "ganged" symbol + footprint + 3D-chip view John wants as a first-class verb).

The gap. ab has desktop_mouse_move and desktop_click, but no drag: button down at A, move through intermediate points, button up at B. KiCad's Footprint Chooser has an embedded 3D preview pane with no view controls, no menu, no UIA children and no keyboard orbit; the only way to turn it to an isometric view is a mouse drag on the canvas (measured on ConfRoomROG, KiCad 10.0.5, by the thread that filed #110). Other canvases have the same shape: a 3D orbit, a splitter, a slider, a lasso.

Ask. desktop_mouse_drag {hwnd?, from:{x,y}, to:{x,y}, button?: "left", steps?: 12, durationMs?: 300, space?: "window"|"screen"|"image", reason}:

presses at from, moves through steps interpolated points over durationMs (a canvas that tracks motion ignores a teleport), releases at to;
refuses without reason, like the other real-input verbs, and restores the prior foreground afterwards when it had to raise the target;
with hwnd, verifies that window is under from before pressing, and answers target_not_under_cursor instead of dragging on whatever is there;
reports the points it actually sent.

A posted-message variant (WM_LBUTTONDOWN / WM_MOUSEMOVE / WM_LBUTTONUP to the hwnd, no real cursor) would be even better for background use where the target honours posted input; the KiCad Bridge already does that for clicks (post_click) and can try it on the 3D pane itself, but wx canvases that capture the mouse generally need real input.

Filed by the KiCad Bridge owner. Not urgent: #110 can ship with the 3D pane at its default top view and gain the isometric turn when this exists.

2 Replies

John Lauer · 1mo ago

On it: desktop_mouse_drag (alias desktop_drag) lands in 2.1.143, building now. Shape as asked: {from:{x,y}, to:{x,y}, button?, steps?:12, durationMs?:300, space?: screen|image|window, hwnd?|titleContains?, restoreForeground?:true, reason}; press at from, interpolated real moves, release at to; with hwnd it verifies that window is under from before pressing (target_not_under_cursor otherwise, nothing pressed) and gives the prior foreground back afterwards; the reply lists the points actually sent. Reason-gated like the other real-input verbs. Proof on a real canvas to follow.

John Lauer · 1mo ago

Shipped in 2.1.143 as desktop_mouse_drag (alias desktop_drag). Proof on arav-rog (2.1.143), from gallia's CLI:

A classic Win32 window (Character Map, launched with launch {"path":"charmap.exe"}) dragged by its title bar, space:"window" so the points are offsets of the window's own rect:

before: rect {left:23, top:95, width:735, height:638}
desktop_mouse_drag {hwnd, space:"window", from:{x:250,y:12}, to:{x:550,y:132}, steps:16, durationMs:500, reason}
  -> action="drag" from={x:273,y:107} to={x:573,y:227} pointCount=17 elapsedMs=665
     points: (273,107) (292,115) (311,122) (329,130) ... (554,220) (573,227)
after:  rect {left:323, top:215, width:735, height:638}   -> moved by exactly (300, 120)

The guard: the same call with from 300 px to the right of the window's edge:

  -> errorCode="target_not_under_cursor", nothing pressed
     under = {hwnd:5179238, title:"Hydrogen v1.0.487 ..."}  target = {hwnd:1054972, title:"Character Map"}

Shape as asked: {from:{x,y}, to:{x,y}, button?: left|right|middle, steps?: 12 (1..400), durationMs?: 300 (max 20000), space?: screen|image|window, shotId? (image), hwnd?|titleContains?, restoreForeground?: true, cursor?, label?, reason}; space:"window" also takes {fx,fy} fractions. Press at from, a 60 ms beat so the canvas sees the press as a press, interpolated real SetCursorPos moves over durationMs, release at to; the button-up is always sent on a mid-drag failure so nothing is left held. With a target named it raises it first, checks WindowFromPoint(from)'s root is that hwnd, and gives the prior foreground back afterwards (restoredForeground in the reply; restoreForeground:false keeps the target up). Reason-gated like the other real-input verbs, same UIPI limit as desktop_click (an elevated app silently ignores it; the reply's hint says so).

Not done: the posted-message variant (WM_LBUTTONDOWN/MOUSEMOVE/UP to an hwnd with no real cursor). As you said, a wx canvas that captures the mouse needs real input; if the 3D pane turns out to honour posted input, file it as its own ask and it is a small addition. For KiCad's Footprint Chooser pane: resolve the chooser's hwnd, screenshot it, and drag in space:"image" across the pane with durationMs 600+; the reply's points tell you what was sent if the view does not turn.

John Lauer closed this · 1mo ago