Index of every page on this site: llms.txt
All skills
vercel-labs avatar

/agent-desktop-windows

@e8934a5
by Vercel Labsvercel-labs/agent-desktop26k stars
1,923
Run it yourself

Windows platform guide for the agent-desktop CLI. Lists per command group what works on Windows and what refuses, the first-contact gotchas (PowerShell ref quoting, minimized windows, headed-only input), and where to find shell surfaces, notifications, the cursor overlay, UIPI elevation, Chromium/Electron settle behavior and troubleshooting. Triggers on: "windows desktop automation", "UIA tree", "automate Windows app", "agent-desktop on Windows", "elevated window input blocked", "E_ACCESSDENIED", or any Windows GUI interaction task.

Use this Skill: https://skilld.dev/gh/vercel-labs/agent-desktop/agent-desktop-windows

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈140 tokens for metadata: the name and description. ≈1.9k when used: this file. ≈7.1k more on demand in 4 files.

Description uses 7.0% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈562 characters, or 7.0%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

agent-desktop-windows

This file covers only what differs on Windows. The observe-act loop, refs, JSON envelope, sessions and tracing are the same on every platform.

Requires Windows 10 1809+ / Windows Server 2019+ (x64 or ARM64). Run it in the interactive desktop session of the signed-in user. Session 0 (services and SYSTEM tasks), Server Core, the secure desktop (UAC prompts, the sign-in screen), a locked desktop and another user's session are not supported: observation, actions and capture there fail or return nothing.

Capability Table

Group Commands Status on Windows
Observation snapshot, find, get, is, screenshot Works. Surfaces are window, focused, a Chromium modal as sheet, and an open application menu as menu
Surfaces list-surfaces Works. Lists window, focused and sheet per process, plus menu with item_count while a menu is open
Shell surfaces open-system-surface Works for start-menu, taskbar, system-tray, system-tray-overflow and action-center, and returns the window snapshot --surface <kind> reads. quick-settings returns INVALID_ARGS naming action-center
Ref interaction click, right-click, type, set-value, clear, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to Works, semantic in headless and --headed mode. right-click and type need --headed and return POLICY_DENIED headless
Multi-click and focus double-click, triple-click, focus Works with global --headed only
Keyboard and mouse press, hover, drag, mouse-click, mouse-move, mouse-wheel Works. Cursor-moving input needs --headed. First Contact covers press --app
Held input key-down, key-up, mouse-down, mouse-up Fails closed on every platform with ACTION_NOT_SUPPORTED. A headless mouse-down or mouse-up returns POLICY_DENIED first
App and window launch, close-app, list-windows, list-apps, focus-window, resize-window, move-window, minimize, maximize, restore Works. First Contact covers the name launch takes
Displays list-displays Works
Clipboard clipboard-get, clipboard-set, clipboard-clear Works; typed text and image content
Wait wait Works for ms, --element, --window, --text, --menu, --menu-closed, --event and --notification
Notifications list-notifications, dismiss-notification, dismiss-all-notifications, notification-action Works over the Action Center. Pass --headed when the center is closed
Cursor overlay cursor-overlay Works. enable returns data.rendered: true only after the renderer acknowledges over its pipe. See references/shell-and-overlay.md
System status, permissions, version, batch, skills, session, trace Works

First Contact

  • Quote every ref in PowerShell. PowerShell reads a bare @token as its splatting operator and deletes the argument before the binary sees it, so set-value @s8f3k2p9:e1 hi fails INVALID_ARGS. Write set-value '@s8f3k2p9:e1' hi. cmd.exe and bash need no quoting.
  • A window behind another window is drivable. A minimized one is not. Every element of a minimized window reports offscreen and ref actions fail. Run restore --app <image> and snapshot again. Do not use focus-window by reflex: it steals the user's foreground.
  • No permission dialog exists. permissions reports automation as not_required. Elevation boundaries are the only access control.
  • Chromium and Electron apps read thin before they settle. Pass a longer --timeout-ms to snapshot instead of concluding the tree is empty.
  • Blocked shortcuts. alt+f4, win+l, win+d, alt+tab, win+r, win+x, win+e, ctrl+shift+esc and their modifier supersets are refused without --force. In combos, cmd and meta are the Windows key, so meta+c is a shell shortcut. Write ctrl+c to copy.
  • type needs --headed. It is keystroke synthesis through SendInput, so a strict-headless type returns POLICY_DENIED. Use set-value, or pass --headed. It ends delivered_unverified (verification_scope: "unavailable") when the field exposes no readable Text selection, and can verify when one is readable: confirm with a fresh get.
  • press --app writes to the foreground queue with no per-process targeting. A headless press to a non-frontmost app fails closed. A non-interactive caller (service, scheduled task, CI job) gets delivered_unverified.
  • launch takes an absolute path or a bare name under System32 or the Windows directory, with or without .exe (notepad or notepad.exe), not a display name. Use list-apps to find a running app.
  • Notifications: list-notifications with the Action Center closed needs --headed. Single-item dismiss-notification and notification-action need --headed and --expected-app or --expected-title for the entry at INDEX.
  • Surfaces: an open or context menu is the menu surface. Windows does not advertise a menubar surface.
  • Commands that raise shell chrome take the foreground. Strict headless refuses them with POLICY_DENIED before anything is raised.
  • Stateful actions run once, then verify by re-reading. delivered_verified means the readback matched. ACTION_FAILED with details.kind: "post_action_verification" means the action landed but the readback differs: read details.post_state and do not retry. A set-value the app reformats reads as failed. See references/troubleshooting.md.
  • Unsigned binary warnings are expected. See references/troubleshooting.md.

What differs on Windows

  • Sandboxes are not available on Windows yet. To run agent-desktop in a container you manage yourself, see agent-browser.dev/desktop/guides/docker-container.

  • Ref actions are semantic UI Automation calls in headless and --headed mode, except right-click, double-click, triple-click, type and focus, which are physical or take focus and need --headed.

  • A ref resolves only when it was the only element in the window its snapshot captured with its role, native id, name and description, and exactly one live element still matches. Uniqueness counts the whole window, not the shallow tree a --skeleton or --max-depth snapshot presents, so a twin below the cut still refuses the ref. Identical twins, such as several identical Delete buttons, are refused with AMBIGUOUS_TARGET, details.kind: identity_not_unique, before anything reaches the app; a live second match is saved_ref_ambiguous, and a removed element is STALE_REF, saved_ref_absent. Take a fresh snapshot after a list changes, and use --headed mouse-click --xy when you must pick one of two identical controls.

  • The cursor overlay plays its per-action travel only for headless semantic actions. A --headed action moves the real cursor instead.

References

  • references/shell-and-overlay.md: open it for open-system-surface, tray and taskbar surfaces, notifications, the cursor overlay, hosted (UWP) window identity and menu detection coverage.
  • references/permissions-and-elevation.md: integrity levels, UIPI, blocked combos, protected processes, the cross-process interaction lease.
  • references/chromium-and-electron.md: settle timing, covered-window hazard, WPF wrong-provider trap, unnamed web content.
  • references/troubleshooting.md: symptom-to-cause map, action verification outcomes, saving a document headless, install warnings.

Read any file from the binary: agent-desktop skills get agent-desktop-windows references/troubleshooting.md.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 alert8d3 checks · Risk SAFE
  • Gen Agent Trust Hub9d

    This skill provides technical documentation for the agent-desktop CLI on Windows. It includes considerations regarding the ingestion of untrusted data from the desktop GUI and instructions for managing Windows security settings to permit execution. These features are documented as part of the skill's intended functionality for desktop automation.

  • Socket8d

    No alerts

  • Snyk9d

    Risk: CRITICAL · 2 issues

Signed by skilld at e8934a5. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 2 days ago.

Activeupdated last week
tags
windows-automation, accessibility, uia, ai-agent, gui-automation, cli
requirements
[
  "agent-desktop"
]

README badge

README badge for vercel-labs/agent-desktop/agent-desktop-windows