Breaking

Windows-Use lets LLMs drive Windows GUI without vision

July 5, 2026 at 13:43 EDT

AI Agents
Open Source
Foundation Models

An open-source AI agent called Windows-Use has been released. Rather than relying on computer vision, it reads structured UI data through Microsoft's UI Automation API, letting an LLM decide on clicks, typing and scrolling to control the Windows GUI.

Open Source · MIT License

Windows-Use: an AI agent that drives Windows without screenshots

Instead of analyzing pixels with a vision model, it reads the application's UI element tree directly through the built-in Windows UI Automation API (UIA) — letting the LLM click, type and scroll from structured data.

0.7–1.7s

Action latency today
target 0.5–0.8s

1M+

Users of the underlying Windows-MCP layer

Vision models needed
use_vision=False by default

Two ways to read the screen

Vision-based agents interpret rendered pixels; Windows-Use reads the structured UI tree — more resilient to layout, font and high-DPI changes.

Vision "Computer Use"

Screenshots + vision model
fragile to fonts / DPI

Windows-Use (UIA tree)

Structured elements, no pixels
lightweight & resilient

How an action happens

UIA reads UI tree → LLM decides action → click / type / scroll → app state updates

Runs on a broad stack

Windows 7 / 8 / 10 / 11 · Python 3.10+
Claude, GPT-4o, Gemini, Groq, Ollama, DeepSeek
Tools: click, type, scroll, shortcut, shell, scrape, desktop

Watch-outs

Legacy Win32 & custom-drawn controls may not expose the UI tree
README: unsafe on a production machine — run in a VM or Windows Sandbox

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.