Vercel Labs has open-sourced agent-desktop, a free tool that allows any AI agent to operate desktop applications, giving coding agents a way to use a computer without relying on screenshots.
Vercel Labs · Open-source release
agent-desktop lets AI agents operate desktop apps by reading their structure, not their screenshots
The free, open-source tool gives coding agents a way to use a computer through the accessibility tree, the structured map of buttons, menus, and text fields that apps already expose to screen readers.
How the agent sees the screen
A structured description replaces pixel interpretation
Step 1
Desktop app
Any application running on the computer
→
Step 2
Accessibility tree
Buttons, menus, and text fields described as structured data
→
Step 3
Agent acts
Identifies what is on screen and interacts with elements directly
Why it matters
Offers an alternative to repeated screenshots that a model must analyze before every action
Could reduce the visual processing required for computer use
Free and open source, lowering the barrier to experimentation
Lets developers add computer use to coding agents they already run
Open questions
No benchmark results or performance comparisons yet
Supported operating systems not detailed
Unclear how many apps expose complete accessibility data
Handling of apps with incomplete or inconsistent labels is unknown
Bottom line
The approach is promising, but its real-world reach depends on testing against the specific apps developers want to automate.
Platform coverage
Which operating systems are supported
App compatibility
Which applications expose complete accessibility data
Label quality
How incomplete or inconsistent labels are handled
Continue reading The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.
Already purchased? Sign in ✓ Signed in — this article isn’t included in your current plan.Unlocking the full article...