Key takeaways
- AI agents can operate desktop apps without guessing from screenshots: each element gets a ref the agent can act on.
- Every action is checked by taking a fresh snapshot, so the agent confirms the result instead of assuming it.
- Actions run headless by default, without moving your pointer or taking focus.
- It is open source under Apache-2.0 and runs on macOS, Windows and Linux.
- Catch: snapshots and screenshots can contain private desktop content, so keep them out of public logs.
What it is
agent-desktop is a Rust command-line tool and C library that lets AI agents read and operate desktop apps through the operating system's accessibility tree. It is open source and the code is at github.com/vercel-labs/agent-desktop.
The problem it solves
Agents that use a computer often work from screenshots and have to guess where a button is. That is slow to debug and easy to get wrong, and the people who feel it are developers building agents that must use ordinary desktop software.
agent-desktop takes a different route. A snapshot returns JSON with a ref for every element an agent can act on, and the agent checks its work with another snapshot, with no pixel matching.
What you get
- Precise targets: each clickable element has a ref, so the agent does not rely on pixel positions.
- Safer retries: stale refs return an error, and the README says to retry only when the error marks it safe.
- Less disruption: actions are headless by default and do not move the pointer or take focus.
- Reach: it runs on macOS, Windows and Linux, and a C library ships for use inside other programs.
The catches
- Permissions: macOS needs Accessibility permission, and Linux needs the user's graphical session with at-spi2-core.
- Uneven support: on Linux, mouse input is X11 only.
- Privacy: snapshots, clipboard reads and screenshots can contain private desktop content, and the optional JEV helper sends screen descriptions to a configured model provider.
- Maturity: the README labels it a Vercel Labs product and makes no stability promise.
- Licence: Apache-2.0, a permissive licence.
What we did not test
We did not install or run agent-desktop, so we cannot say how reliable it is on real apps or on any one operating system. The README gives no benchmarks, and every claim here comes from the README or the GitHub API.
Verdict
Editors' view: this is for developers building agents that must use desktop software and want structure rather than screenshots. It is worth a trial on a spare machine, but treat it as an early Labs project and keep its output out of shared logs.
Sources
Need help adopting this, or have a software project in mind?
Tell us what you are building. The initial consultation is free. Email [email protected] or follow @zolnaltd.
Start a conversation →