agent-win

Native desktop UI automation for AI agents — the OS-level complement to a browser driver. Drive any Windows app through UI Automation (UIA) with a tight snapshot → ref → act loop: snapshot the on-screen accessibility tree into ref-tagged interactive elements, then click/fill/toggle them by ref — through the control's own UIA pattern, so the mouse never moves. Built to be driven by AI agents (machine-readable --json, single self-contained commands) and pleasant for humans too. (Same loop shape as agent-browser, but for the OS.)

agent-win owns the native desktop: app windows, menus, and above all native OS dialogs — permission modals, file pickers, UAC-style confirmations — that a browser's CDP/DOM can't see. It is deliberately not a web page/tab reader: for page content, tabs, and the DOM, use a browser driver (e.g. agent-browser over CDP). The two compose — see Browsers: launch + hand off below.

$ agent-win snapshot --window "Calc"
window: @w2 "Calculatrice" (ApplicationFrameHost.exe) [focused]

@e24 [Button] "Est égal à"  #equalButton
@e30 [Button] "Cinq"        #num5Button
@e12 [Text]   "L’affichage est 0"  #CalculatorResults

$ agent-win click @e30          # invoke num5 — no mouse movement
clicked @e30 "Cinq" (via invoke)
$ agent-win read @e12
L’affichage est 5

Why UIA beats the OCR/pixel path

Raw MoveMouse(x, y); Click() off a screenshot is brittle: you eyeball scaled pixels, the click misses when a reflow moves things, and controls that pixels can't see (native permission dialogs, menu items, off-DPI buttons) are unreachable. agent-win instead:

• Addresses elements by identity, not coordinates. Every action re-resolves the ref to its live control by RuntimeId → AutomationId → structural path, so it survives relayout, scrolling, and even the target app restarting.
• Acts through UIA patterns — no mouse, no focus-stealing. click fires InvokePattern.Invoke() (falling back to LegacyIAccessible.DoDefaultAction, then SelectionItem.Select). The window need not be focused or on top, and the cursor stays exactly where the user left it. This is the concrete win: it drives the "Autoriser" browser-permission dialog that OCR can't reliably hit.
• AutomationId targeting is locale-independent. On a French Windows the digit is named "Cinq" but its id is #num5Button — find "num5" works in any language.

Browsers: launch + hand off (not page-driving)

agent-win's browser-adjacent job is exactly the part a browser driver can't do itself: launch the browser and click the native OS dialogs around it. The canonical case is enabling remote debugging — Edge/Chrome raise a native permission modal ("Autoriser this app to…" / "Allow") that lives in the window chrome, invisible to CDP and the DOM. agent-win finds and invokes it regardless of which window is focused:

$ agent-win find "Autoriser"     # searches ALL windows — found even if a terminal is foreground
@w7 "Activer le débogage à distance ?"
  @e1 [Button] "Autoriser"
$ agent-win click @e1            # InvokePattern — accepts it, no mouse, window need not be focused
clicked @e1 "Autoriser" (via invoke)

wait: catching a dialog that only exists while something waits on it

A native permission modal is a race. It appears when a client starts connecting and it is gone once the client gives up, which for playwright-cli is 30 seconds. Polling find in a shell loop loses that race, because an unscoped find walks every window and takes about 9 seconds.

ref=$(agent-win wait "Autoriser" --actionable --timeout 45 --json | jq -r '.matches[0].ref')
agent-win click "$ref" -q

Real output, against a pending CDP handshake:

wait returned ref: e511
clicked @e511 "Autoriser" (via invoke)
RESULT: HTTP/1.1 101 WebSocket Protocol Handshake     <- the socket that was blocked

--gone waits for the opposite, which is how you confirm a dialog actually closed. Timeout exits 1 with code=timeout, so an agent branches on the exit code instead of parsing prose.

daemon: stop paying COM startup on every command

Each command is its own process and spends about 550ms importing uiautomation and starting COM before it learns anything. agent-win daemon start pays that once and serves commands over a loopback socket. Measured on windows:

	cold	warm
first call	2276 ms	663 ms
steady state	1023 ms	505 ms

Two deliberate limits. It caches no UI trees, because a cached tree is a stale tree. And it removes startup, never walking — an unscoped find stays dominated by heavy windows, so scope with --window when you can.

The client falls back to running in-process whenever the daemon is missing, unreachable, or a different protocol version, so it is always an optimisation and never a dependency. Set AGENT_WIN_NO_DAEMON=1 to bypass it.

Worked example: attaching to a browser no CDP client can reach

Enabling remote debugging from chrome://inspect produces an endpoint that no CDP client can attach to on its own. It serves no /json/* routes, so auto-discovery fails, and every websocket upgrade hangs while the browser shows an "Autoriser"/"Allow" button in its own window chrome. The client blocks on the handshake; the thing that unblocks it is a click CDP cannot see. Clients with a short handshake timeout (playwright-cli: 30s) give up before a human reaches the mouse.

tools/cdp_attach.py closes the loop: it opens the websocket itself and, while the handshake is pending, drives agent-win to find and invoke the modal — then speaks CDP over the approved socket.

$ python tools/cdp_attach.py
endpoint: ws://127.0.0.1:52245/devtools/browser/9a086eef-…
  approve: clicking e366                       # agent-win, no mouse, Edge need not be focused
connected: HTTP/1.1 101 WebSocket Protocol Handshake
5 open page(s):
  'Your Repositories'   https://github.com/…
  'flashmind'           https://master-it-yassi.vercel.app/courses

--demo opens a tab and walks it through three sites with pauses, reading each real page title back over CDP — a watchable end-to-end check that the whole chain works:

$ python tools/cdp_attach.py --demo
  approve: clicking e372
connected: HTTP/1.1 101 WebSocket Protocol Handshake
opened a tab (target D6E31DE8...)
  https://example.com             ->  title read back: 'Example Domain'
  https://news.ycombinator.com    ->  title read back: 'Hacker News'
  https://developer.mozilla.org   ->  title read back: 'MDN Web Docs'

The script is stdlib only — a raw socket for the websocket handshake and frame masking, subprocess for agent-win. No pip install, no CDP library.

Two gotchas that cost real time. The uuid on line 2 of DevToolsActivePort changes every time the toggle is switched — a stale uuid times out exactly like a rejected one. And sending an Origin header turns the hang into a 403 naming --remote-allow-origins, which is a red herring: omitting Origin is what works.

Once attached, hand the page / tab / DOM work to agent-browser (or any CDP client). agent-win intentionally does not enumerate browser tabs or read page content — that is the browser driver's lane. Keep the split: agent-win = native windows + OS dialogs; the browser driver = the web page.

Requirements

• Windows (UIA is Windows-only) and Python ≥ 3.9.
• The uiautomation package (installed automatically): pip install uiautomation.

Install

# from the repo root
pip install -e .          # registers the `agent-win` (and short `awin`) commands
agent-win help

Or run without installing, from the repo root:

python -m agent_win help

(agent-win.cmd / agent-win.ps1 shims are provided for PATH use.)

The loop

1. agent-win windows — list top-level windows as @wN (persisted, so later commands can --window @wN).
2. agent-win snapshot [--window @wN|title] — print the interactive elements as @eN.
3. Act on a ref: click @e3, fill @e7 "text", toggle @e5, key "{Enter}".
4. The action prints the resulting state itself. Snapshot again only to change window.

Every action answers "what happened"

An action used to report the mechanism it fired (clicked @e1 (via invoke)) and stop there. That proves a UIA pattern ran. It proves nothing about whether the right control was hit or whether anything changed. So actions now print the state of the window they acted on, by default.

The view follows the element, not the foreground. Driving a background window shows that window, which is the whole point of UIA.

	prints	use when
(no flag)	full state of the window acted on	default — safe, needs no memory of the last view
--diff	only what appeared, changed, or went away	terse — assumes you saw the last view
-q	just the action line	cheapest — chaining a sequence you already trust

Long views end with a one-line reminder that those flags exist. It fires only past 40 elements, so it stays a signal instead of boilerplate. --then is still accepted and now does nothing.

To locate a control (e.g. a dialog that popped up on a window you aren't focused on), agent-win find "text" searches all windows and reports each hit's @wN; scope with --window.

Refs are stable

A ref number is bound to the control's identity, not to its position in the last listing. @w3 is the same window after the Z-order changes, and the same button comes back as the same @eN on every later find. Numbers are never reused, so a ref can go stale but can never come back pointing at a different control. Refs also outlive a change of scope: snapshot window A, snapshot window B, and A's refs still act.

The trade: numbers are not sequential down a listing. Read the ref off the line rather than assuming @e1. agent-win reset clears the numbering.

Commands

Command	Purpose
windows	List top-level windows as @wN: title, process, focused.
snapshot [query]	Interactive elements of the scope as @eN; query filters by name/AutomationId.
find "text" [--type T] [--actionable] [--window @wN]	Substring search over all windows (or one via --window) by name/AutomationId (and control type); each hit reports its owning @wN. --actionable keeps only real controls; actionable hits rank above text-only matches. Empty "text" + --type lists every control of that type.
read <@eN> [--full|--range A:B]	Element text/value, middle elided unless --full/--range.
tree <@eN> [--depth N]	Indented subtree under one element (inspection; assigns no refs).
click|invoke <@eN>	InvokePattern → LegacyIAccessible.DoDefaultAction → SelectionItem. No mouse.
fill|settext <@eN> "text"	ValuePattern.SetValue → focus + clear + type fallback.
focus <@eN>	SetFocus the element.
toggle <@eN>	TogglePattern.Toggle (checkboxes / switches).
expand <@eN>	ExpandCollapsePattern — expands if collapsed, else collapses.
select <@eN>	SelectionItemPattern.Select (list items / tabs / radio).
type "text"	Type literal text into the focused element (metacharacters escaped).
key "keys"	Raw SendKeys to the focused element: {Ctrl}a {Enter} {Delete} {Tab} …
wait "text" [--gone] [--timeout S]	Block until a control appears (or disappears). Exits 0 with the hits, or 1 with code=timeout.
daemon [status|start|stop]	The warm UIA process other commands use when it is running.
reset	Clear ref numbering and the ref index; the next listing starts at @w1 / @e1.
help [cmd] | docs	This help / the extended agent reference.

Flags: --window <@wN\|title>, --all (include non-interactive named elements), --depth N, --type <ControlType>, --actionable (find: real controls only), --full / --range A:B, --json, --then. -h/--help is accepted everywhere.

Driving agent-win (gotchas)

A field guide for agents driving the CLI:

• Pass refs WITHOUT the @ from PowerShell. Output prints refs as @wN/@eN, but PowerShell eats a bare @e5 as splatting syntax, so agent-win invoke @e5 arrives empty. Use the no-@ form — invoke e5, snapshot --window w6 — which works in every shell. Both forms are accepted; the no-@ form is shell-proof. (A missing/blank ref now says exactly this, with the fix.)
• find is a substring match — narrow it. find "text" matches any control whose Name or AutomationId contains the text, including labels and document text (e.g. an editor showing the word). Add --actionable to keep only real controls (button/checkbox/…), or --type Button to filter by control type. Actionable hits rank above text-only matches, so the first line is the useful control. Empty "text" + --type lists every control of that type; use an AutomationId (find num5) for locale-independent targeting.
• Cross-window find walks every window — scope it when it's slow. Browser trees are heavy; a window that takes ≥1.5 s to walk prints note: … large UI tree (Ns) — scope with --window … (stderr), and a total 25 s deadline stops the search and reports what it skipped. Pass --window <@wN|title> to search just the window you care about.
• --json errors are structured. Every failure prints {"error","hint","code"} on stdout (non-zero exit) with a stable code you can branch on (stale_ref, no_window, not_invokable, missing_ref, access_denied, …); in human mode the same error: <what> <hint> goes to stderr, so --json stdout stays pure and parseable. Every element also carries an actions list (invoke/fill/toggle/expand/select) telling you which verb will work before you try.
• Elevated windows read empty. A non-elevated agent-win can't see an admin app's tree; an empty snapshot explains this (run agent-win elevated to match) rather than looking like a plain miss.

Refs survive across commands (ref stability)

Each agent-win call is its own process, so a ref can't be a live UIA object. snapshot and find write a re-resolution key per element ref (and windows/snapshot/find write one per @wN) to %LOCALAPPDATA%\agent-win\refs.json (override the dir with AGENT_WIN_STATE). The next command reloads that key and re-finds the current live control, in this order:

1. Anchor window by native handle, falling back to title.
2. Structural path (child-index path from the window) + control-type/AutomationId match — fast.
3. Full re-walk matching RuntimeId (exact within a session), else control-type + AutomationId/name, picking the candidate closest to the old center.

RuntimeId is exact per session; AutomationId is stable across sessions when the app sets it; the structural path + name is the universal fallback for apps that expose neither. Ref lifetime: @eN are valid until the next snapshot/find; @wN until the next window enumeration (windows, or a foreground/title snapshot/find) — a --window @wN command reuses them without reassigning. A target that's gone returns a clear stale ref … — re-run snapshot error (exit 1) telling the agent to re-snapshot.

Token efficiency (compression)

• Interactive-only by default. Snapshots list buttons, edits, checkboxes, menu items, tabs, list items, etc. — and prune offscreen subtrees (hidden tabs/menus). --all widens to every named element; --depth N bounds the walk.
• Long text is elided in the middle: values are previewed (~48 chars) and read keeps a leading + trailing slice with …[N chars elided — read @eN --full or --range A:B]…. Expand on demand with read @eN --full or read @eN --range 0:2000.
• --json is compact; the human view is the compressed indented one. stdout is pure data; diagnostics go to stderr.

Scope

snapshot defaults to the foreground window, excluding this CLI's own console. When you drive from a terminal, that terminal is often the foreground window — run windows then snapshot --window @wN (or --window "Title") to target the app exactly. find, by contrast, searches every top-level window by default — so a dialog on a background window is still found — and --window narrows it to one.

Limitations

• Windows-only — UIA has no macOS/Linux equivalent.
• Unnamed / custom-drawn controls. Apps that skip UIA (some Electron/game canvases, raw GDI) expose few named elements; --all helps, otherwise those regions are opaque.
• Elevated windows. A non-elevated agent-win cannot read or drive an elevated (admin) app's tree — run agent-win elevated to match, or it returns an empty subtree.
• type/key send to whatever is focused — focus <@eN> (or fill) first.

Alternatives

Built on uiautomation (yinkaisheng) for its clean tree walk and pattern wrappers. For heavier needs consider pywinauto (higher-level app/window driving, backends uia+win32) or FlaUI (.NET, the most complete UIA surface). agent-win keeps a deliberately small, agent-shaped CLI on top of the same OS API.

License

MIT