Computer use

Computer use is a class of agentic AI where the model sees a screenshot of a real desktop or browser, decides where to click and what to type, and emits low-level UI actions (mouse_move, click, type, scroll). It's how you automate software that doesn't have an API: legacy enterprise apps, finicky web forms, anything behind a login wall. Anthropic shipped Computer Use with Claude 3.5 Sonnet in October 2024, the first frontier model with this capability. OpenAI followed with Operator in early 2025. Google, Adept, and others have variants. By 2026 every major lab has a computer-use offering, and the loop is roughly the same: take a screenshot, ask the model what to do next, execute that action in a sandbox VM, repeat. It works — but it's slow, expensive, and brittle compared to API-based tools. A task that takes a structured tool call 200ms takes a computer-use agent 30+ seconds because of screenshot tokens and reasoning per step. Pixel-level UIs change. Pop-ups break flows. Captchas exist. Rates of unattended completion on real-world tasks are still in the 40-70% range on benchmarks like OSWorld and WebArena, depending on difficulty. Where it shines: back-office automation where the alternative is hiring someone, demos that need to look like a human is using the app, and bridging to systems with no API. Where it doesn't: anything time-sensitive, anything you can call a real API for, anything that needs to be 99.9% reliable.