Anthropic released computer use in public beta with Claude 3.5 Sonnet. The model could use screenshots and request virtual mouse and keyboard actions through the API.
Computer Use
Also known as: Computer-Using Agents · GUI agents · Browser agents
Plain English
Computer use lets an AI model work through the visible controls of a browser or desktop application. It can read what appears on screen and ask a tool to click, type, or scroll.
Technical Overview
A computer-use harness presents screenshots or structured interface observations to the model, executes authorized actions, and returns updated state. This supports software that lacks a suitable API, but changing layouts, ambiguous visual state, and multi-step errors can reduce reliability. Human confirmations and isolated environments help control consequential actions and access.
Timeline (4)
2024
Release2025
ReleaseOpenAI introduced Operator as a research preview of a model trained to act on visual observations. Its system card described safeguards and evaluated end-to-end task completion in simulated environments.
- Release
Google introduced a specialized Gemini 2.5 Computer Use model in API preview. It returned proposed interface actions for the caller to execute against a browser or application.
2026
ReleaseOpenAI introduced native computer use in its general-purpose GPT-5.4 model. A single model could reason over a task and produce mouse or keyboard actions from screenshots.
See also (7)
Open full graphComputer Use