Index of every page on this site: llms.txt
All skills
vercel-labs avatar

/agent-desktop-macos

@e8934a5
by Vercel Labsvercel-labs/agent-desktop26k stars
1,923
Run it yourself

macOS platform guide for the agent-desktop CLI. Use for desktop automation on macOS: granting Accessibility and Screen Recording permission (TCC), what headed mode does on macOS (physical-first click, type and scroll), surfaces (menu, menubar, sheet, popover, alert, dock, menu-bar-extras), menu bar apps and system panels with no window, Notification Center, Chromium and Electron apps including launch --cdp, and fixing PERM_DENIED, empty accessibility trees, ACTION_FAILED and APP_UNRESPONSIVE.

Use this Skill: https://skilld.dev/gh/vercel-labs/agent-desktop/agent-desktop-macos

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencestroubleshooting.md

≈1.7k tokens on demand. Your agent reads this file only when SKILL.md points to it.

macOS troubleshooting

Each entry is a symptom, its cause, and the fix. Read error.code, error.suggestion and error.disposition in the JSON envelope first.

PERM_DENIED

Symptom. A command returns PERM_DENIED, or permissions shows denied.

Cause. The app that launched agent-desktop has no Accessibility permission (or Screen Recording, for screenshot and session start --screenshots). The grant belongs to the launching app, not to the shell. A new launcher, such as a different terminal or an IDE, needs its own grant.

Fix. Follow "First contact" in the macOS skill: permissions --guide, turn on the launching app (and the agent-desktop binary if it is listed on its own) in System Settings > Privacy & Security > Accessibility or Screen Recording, then run agent-desktop permissions to confirm granted. The launching app does not need a restart.

The guide offers the app that macOS charges the permission to, which is the app that started agent-desktop's process tree. Under tmux, that is the app that started the tmux server, which can differ from the terminal you attach from. Under an SSH session, no such app exists, so the guide offers the agent-desktop binary. macOS 27 and later name the Accessibility list Device Control and Data Access.

For Automation, open Privacy & Security > Automation and allow the launching app to control System Events.

Empty or sparse tree

Symptom. snapshot returns few nodes, or no refs.

Cause. Some apps expose little accessibility data: custom-rendered UIs, games, some creative tools, and parts of web views. Chromium and Electron apps need web accessibility enabled in the renderer.

Fix.

  1. Check permissions first (see PERM_DENIED).
  2. Remove -i to see non-interactive nodes. Raise --max-depth.
  3. Run list-surfaces --app "App". The content may sit in a sheet, popover or menu surface.
  4. Retry the snapshot once. A renderer needs a short settle time after accessibility is enabled.
  5. Use screenshot as a visual fallback (needs Screen Recording).
  6. For a fresh Chromium app, use launch "App" --cdp and a CDP client for the web contents.

Slow or truncated snapshot

Symptom. A snapshot is slow, or returns "complete": false with "truncated": true.

Cause. Large apps (Xcode, Safari with many tabs) have deep trees. A truncated snapshot ran out of its observation budget.

Fix. Use find when you know the target. Otherwise use snapshot --skeleton -i --compact, then snapshot --root @<snapshot_id>:eN to drill down. Add --window-id to limit the window.

ACTION_FAILED

Symptom. A ref action returns ACTION_FAILED.

Cause. The element is disabled, the app is busy, or the element does not support the action. The semantic chain tries only the actions the element advertises, then stops.

Fix.

  1. Run is @ref --property enabled. Wait or fix the state if it is false.
  2. Run get @ref --property bounds and check that the element is on screen.
  3. If physical input is right for the task, retry with --headed.
  4. As a last step, use the keyboard: focus @ref, then press return.

POLICY_DENIED

Symptom. A command returns POLICY_DENIED.

Cause. The semantic path could not finish, and the physical path is blocked because the command ran headless. Hover, drag, double-click, triple-click and cursor-moving mouse commands always need --headed. Coordinate right-click is also blocked headless.

Fix. Add the global --headed flag only if physical input is what you want. Example: agent-desktop --headed mouse-click --xy 640,400 --button right. Take coordinates from get @ref --property bounds.

APP_UNRESPONSIVE after right-click

Symptom. right-click returns APP_UNRESPONSIVE with delivery_uncertain and retry: unsafe.

Cause. AXShowMenu can enter modal menu tracking and return kAXErrorCannotComplete after the menu opened.

Fix. Do not retry. Run snapshot --app "App" --surface menu -i. If the menu is there, use it. If not, run list-surfaces --app "App". For a combo box or menu button, use select @ref "Option".

select does not find the menu

Symptom. select fails to confirm the choice.

Cause. select proves success by a closed-to-open menu change. An already open menu hides that change.

Fix. Dismiss the open menu with press escape, then run select again.

APP_NOT_FOUND or WINDOW_NOT_FOUND

Fix. Run agent-desktop launch "App Name" (display name or bundle ID). Then run list-windows --app "App". A full-screen app still appears in list-windows, but its bounds equal the screen and its state can lag behind animations, so use wait.

Notification Center

Symptom Cause Fix
POLICY_DENIED when listing: the center is closed The command is headless and cannot open the center Add --headed, or open the center yourself
dismiss-notification returns ACTION_FAILED No dismiss path worked on that item See the chain below, then list again
NOTIFICATION_NOT_FOUND The index moved after a dismiss Run list-notifications again
Banner gone before you list it Banners are transient Use wait --notification --app "App"
A dismissed item is replaced by another Notifications from one app are stacked Dismiss again. dismiss-all-notifications may need more than one round
Calendar widget stays It is a system widget, not a notification Expected

The dismiss chain runs in order: the AXDismiss and AXRemoveFromParent actions, then a close button named close, clear or dismiss, then hover to reveal a hidden close button and press it. AXPress is never used, because it opens the source app and does not dismiss.

Keyboard shortcut does nothing

Cause. The shortcut used ctrl where macOS uses cmd, or the keys went to the wrong app.

Fix. Use cmd+.... To send keys to one app without focusing it, use press cmd+s --app "App".

Sandbox answers needs_setup

Symptom. agent-desktop sandbox status <name> answers needs_setup for a sandbox that worked before.

Cause. The guest agent, agent-desktop-guest, is not running in the guest, or Accessibility is off for it. The agent starts with each login to the guest account, so the guest needs automatic login.

Fix. Run agent-desktop sandbox stop <name>, then agent-desktop sandbox start <name> --window, and in the guest turn on automatic login, run sh "/Volumes/My Shared Files/guest-setup.sh" in Terminal, and switch on agent-desktop-guest in System Settings > Privacy & Security > Accessibility. Then check status on the host. A sandbox set up by agent-desktop 1.3.0 needs this once.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 alert1d3 checks · Risk SAFE
  • Gen Agent Trust Hub2d

    This skill provides a comprehensive guide for automating macOS desktop tasks using the agent-desktop CLI. It includes instructions for managing necessary system permissions and troubleshooting common GUI automation challenges.

  • Socket1d

    No alerts

  • Snyk2d

    Risk: CRITICAL · 3 issues

Signed by skilld at e8934a5. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 2 days ago.

Activeupdated 3 days ago
tags
macos-automation, accessibility, ax, tcc, ai-agent, gui-automation, cli
requirements
[
  "agent-desktop"
]

README badge

README badge for vercel-labs/agent-desktop/agent-desktop-macos