tldr; Computer-use models have gotten surprisingly good at clicking buttons and interfacing with your computer. The next leap comes from improvements to maintain state, notice changes, choose between pixels versus API, PoW, and recovering from weird states.The model is already smart enough. The harness around it still sucks.

There's something humiliating about watching a frontier model use a computer. The model can solve Olympiad problems, chunk and analyze large legal contracts, and explain why distributed systems could deadlock.

Then it tries to close a pop-up. It sees a button. It reasons correctly about it, produces sensible coordinates for this action. However this pop-up moves 200 ms later. The model inadvertently clicks an advertisement.

This is the strange state of CUA's in 2026. On established desktop benchmarks, some frontier systems have gone from functional to human-level in around 2 years. But workflows that resemble actual work expose a very different capability frontier. On long, stateful tasks, agents still lose track of attachments, forget constraints, miss information that arrives midway through the task, and fail to verify whether the final result is actually correct. OSWorld-2.0 showed this gap where only $20.6\%$ of the workflows were completed e2e.

My working thesis is that computer-use models are already good enough to unlock a much larger class of useful systems, but we are managing them through an impoverished interface: a screenshot, text history, and a loop that says where do I click next.

benchmark hardening#

In 2024, the original OSWorld benchmark found that humans completed $72.36\%$ of its desktop tasks, while the strongest evaluated model complete only $12.24\%$. The agents struggled with basic visual grounding and software knowledge. They could not reliably determine how common applications worked. OpenAI's first CUA raised this to $38.1\%$ in early 2025. By March 2026, GPT-5.4 reached $75.0\%$ on OSWorld-Verified, slightly above its reported $72.4\%$ human baseline. The same model reached $67.3\%$ on WebArena-Verified and $92.8\%$ on Online-Mind2Web.

OSWorld2.0 asked the agent to perform a $\sim 2$ min desktop task, completing workflows that skilled humans take about $1.6$ hours. The average agent trajectory went from $30$ actions to hundreds. Most of these involved multi-application, changing context windows, hidden states, and intermediate decisions. At the benchmark release, Opus 4.8 achieved $20.6\%$ completion and $54.8\%$ partial credit scoring.

A language model that only produces text is constrained by the information and actions that have been placed in the context window. CUA can open source documents, inspect the application, run the code, read the error, compare the the results, and check if modification worked. It turns the "brain" into a system that can do real work. Computer-use is a universal compatibility layer. Applications that don't have API can be interfaced from their GUI and traditional CLI tools can be interfaced with a simple tool. A GUI agent can theoretically interact with all of them without making MCP servers for each application that needs to be surfaced.

trade-off of pixels#

Computer pixels are the closest software to an universal API. But they also suck. Pixels don't have types and "interface-able" DOM needs to be inferred. Universal compatibility is valuable enough that computer-use will matter even when it is slower and less reliable than direct tool integration.

This can be used for narrow systems that use visual interactions to close specific loops that are currently left to humans like:

• Testing the GUI interface produced by coding agents
• Migrating between legacy systems
• Processing repetitive workflows
• Automating applications that cannot be modified or integrated directly

GUI testing#

One of the clearest near term applications that is determinant is testing interfaces produced by coding agents. Given that coding agents rapidly generate components and modify logic. Visual verification remains manual. My project inspector is an example of a system to make this more automated. It uses CUA to test web, electron, Android, and iOS interfaces. Instead of assigning one agent to go through the entire application, it maps the interface to regions, assigns separate agents to test them and returns structured findings to the user. The week after, Cognition released testing and video recordings similarly to my system. $\text{generate code} \rightarrow \text{launch app} \rightarrow \text{explore UI} \rightarrow \text{reproduce failure} \rightarrow \text{patch code} \rightarrow \text{reverify}\text{generate code} \rightarrow \text{launch app} \rightarrow \text{explore UI} \rightarrow \text{reproduce failure} \rightarrow \text{patch code} \rightarrow \text{reverify}$ UI testing is particularly good because the environment is sandboxed and reset. Thus, incorrect actions are usually low risk. Failures are useful to collect and system can preserve traces. Inspector demonstrates the broader architecture computer-use systems need.

pixels and tool use#

This field is moving towards the ideas that MCP should be combined with computer-use. OSWorld-MCP evaluates agents that can choose between GUI actions and 158 tool use across seven applications. This improved o3 from $8.3\%$ to $20.4\%$ for a 15-step budget.

This improvement shows that direct tools can remove visual errors. A script can transform a spreadsheet more reliably than dozens of mouse actions. An API can retrieve structured records without scrolling through a dashboard. The GUI remains valuable because it works when no cleaner interface exists.

improvements around systems#

The systems should need to be built around 5 main capabilities:

• Persistent State: Agents needs to track files, attachments, completed steps, and current state better. The system should know what it has done but what remains true after each action
• Change Detection: Systems should notice continual changes. CUA should be event-driven over screenshot driven
• Dynamic Selection: Agents should decide each steps to use GUI, an API, DO, filesystem, or scripting. Pixel/visuals should be a fallback. A system that routes through the most reliable interface for an action is best.
• PoW: Systems need to track their works and produce evidence that an intended state was reached. Every important action should end with a verifiable condition
• Recovery: Real software produced expired sessions, stale pages, and partial completion. Reliable agents need recovery policies, detecting when the state is not reached to replan, rollback, or diagnose the issue.

References#

Tianbao Xie et al. “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.” NeurIPS 2024.
https://arxiv.org/abs/2404.07972

OpenAI. “Computer-Using Agent.” January 23, 2025.
https://openai.com/index/computer-using-agent/

OpenAI. “Introducing GPT-5.4.” March 5, 2026.
https://openai.com/index/introducing-gpt-5-4/

XLANG Lab and collaborators. “OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks.” 2026.
https://arxiv.org/abs/2606.29537

Wesley Lu and Tiffany Yung. “Inspector: Multi-Agent Computer-Use QA.” Devpost, UC Berkeley AI Hackathon 2026.
https://devpost.com/software/inspector-23ser6

Cognition. “Testing & Video Recordings.” Devin Documentation.
https://docs.devin.ai/work-with-devin/testing-and-recordings

Hongrui Jia et al. “OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents.” 2025.
https://arxiv.org/abs/2510.24563

Pascal J. Sager et al. “A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions.” 2025–2026.
https://arxiv.org/abs/2501.16150