OpenAI has introduced “computer use” capabilities to ChatGPT, allowing the AI to control a user’s mouse and navigate desktop applications. According to recent tests by Ben Patterson of Foundry, the feature enables the AI to interact with graphical user interfaces (GUIs) to perform tasks such as playing chess and writing in a digital journal on macOS.
The update arrives as part of a “super-app” release that includes ChatGPT Work—an agent capable of accessing local files to generate reports—and a built-in Codex coding harness.
For users to enable these features, the software requires specific system permissions. According to Patterson, the app requests the ability to record the screen so the model can see the desktop environment, as well as explicit permission to interact with individual applications. Users are presented with “Always allow,” “Allow this conversation,” or “Deny” options for these permissions.
Testing ChatGPT Computer Use With macOS Chess
In a practical test conducted on a Mac mini, Patterson utilized the GPT-5.6 Sol model in “Work” mode to play a game of chess. The AI successfully launched the macOS Chess app in less than 30 seconds, indicated by a purple screen-sharing icon in the top-left corner of the window.
The AI demonstrated the ability to recognize and counter specific chess strategies. According to the test results, GPT-5.6 Sol identified the “Winawer variation” opening gambit and executed eight moves in under three minutes.
ChatGPT had no trouble moving the pieces in the macOS Chess app.Ben Patterson/Foundry
Comparative Performance Against Claude
Computer use functionality is not exclusive to OpenAI; the Claude desktop app has supported it for months. Patterson compared the performance of the new ChatGPT model against the Fable 5 model using the same chess application.
While earlier versions of Claude reportedly struggled with the graphical 3D interface of the chess app, the Fable 5 model performed similarly to ChatGPT. During the test, Claude successfully detected an “Italian Game” opening and adjusted its strategy accordingly, moving pieces with the same efficiency as the OpenAI model.
Claude’s a lot better at using the Mac’s Chess app than it was a scant four months ago.Ben Patterson/Foundry
Beyond Gaming: Automating Desktop Workflows
The utility of computer use extends to productivity apps. In a separate test, ChatGPT was prompted to open the macOS Journal app and create a new entry. The AI navigated the interface, started a new post, and wrote a reflective entry about its own “curiosity” and the ritual of writing on a blank page.
ChatGPT had no trouble navigating the Mac’s Journal app, cheerfully spilling its thoughts into a new entry.Ben Patterson/Foundry
OpenAI has suggested several developer-focused uses for this technology. These include testing desktop app interfaces, churning through repetitive tasks like trying to reproduce bugs, and accessing app settings that can only be accessed via a graphical interface.
Security Risks and Privacy Concerns
Patterson noted that while the AI could potentially organize a Notes app or balance a checkbook in Quicken, the privacy risks associated with letting an agent “dive into sensitive files” remain a concern. It’s a good idea to close any windows with sensitive information before letting ChatGPT loose on your desktop.