Computer Use

An agent capability to operate a computer interface directly — clicking, typing, reading screens — instead of calling APIs.

Computer use is the agent’s ability to drive a graphical interface the way a human would: take a screenshot, identify UI elements, click, type, and observe the result. Anthropic released a computer-use beta in October 2024; OpenAI followed with Operator; Google with similar capabilities in Gemini.

The appeal is universal coverage — any software with a UI is reachable, even legacy systems without APIs. The cost is fragility: pixel coordinates change with screen size, modals interrupt, animations confuse the model. Benchmarks like OSWorld measure success on real desktop tasks; as of early 2026, frontier models score around 30-40% on long-horizon tasks, with significant room to grow.