Benchmark Almanac

Computer use

Models operate a computer or phone through its screen. We count 149 benchmarks here. A benchmark belongs here when environment includes desktop apps, browser or phone apps, and input includes screens.

How fast it grows

31 were released since July, more than the 30 in the first half of 2026.

  1. Earlier 5
  2. 2024 H1 16
  3. 2024 H2 11
  4. 2025 H1 21
  5. 2025 H2 23
  6. 2026 H1 30
  7. 2026 H2 31

How much is still open

No model has passed 85% on 86 of the 104 benchmarks here that score in percent, 83% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
Browser Use LinkedIn Agents answer plain-language questions about LinkedIn using recorded page snapshots. The leaderboard reports correctness, cost, and time per task. Everyday Nov 2026 97.9 Not in cards
FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. University Nov 2026 74.5% Not in cards
Kaeru Flip (paprika3d) Agents play an 86-note rhythm game from browser pixels. They must finish the song with at most one miss and leave autoplay off. Everyday Nov 2026 Not in cards
WeChat Pay Physical Client Evaluation Agents operate the WeChat Pay client on physical phones. The team scores execution paths, use of supplied knowledge, acceptance checks, and account creation accuracy. University Nov 2026 Not in cards
ArenaWebdevCorrectness Judges inspect generated applications and predict whether their features work. Human labels provide the reference answers. University Oct 2026 81.6% Not in cards
BrowseSafe-Adapt Browser agents face adaptive prompt injections across 1,024 tasks. The score is attack success within 64 attempts. University Oct 2026 30.7% Not in cards
CLASHarness Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment. Everyday Oct 2026 28.7% Not in cards
Gym-Anything-RT Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time. Everyday Oct 2026 9.00% Not in cards
Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. Expert Oct 2026 In use
Realtime-Bench Agents act in 50 live game and web tasks where a late action fails: 30 web tasks (travel, shopping, services and security) and 20 games. The environment keeps moving while the model thinks. Three clocks separate speed from skill: Wall Clock runs on real time, Paused Clock stops while the model thinks, and Token Clock advances with each generated token. Scores are task success rates over three runs. Everyday Oct 2026 42.7% Not in cards
ReTrace Computer agents recover from injected errors in desktop tasks. Replaying the remaining actions checks whether the recovered state produces the original successful outcome. Everyday Oct 2026 63.6% Not in cards
Web Clone Bench Agent gets a running reference app (Gmail, Jira, Figma, etc.) and 8 hours to build a clone. Expert-written workflows are replayed as end-to-end tests. Research frontier Oct 2026 62% Not in cards
WildAWM validation Agents complete 33 held-out computer-use tasks in generated software environments. Harbor packages the validation split separately from the training tasks. Expert Oct 2026 22.3% Not in cards
AnyAppBench Agents repeat the same task in different Android apps, with and without suggested steps. Each app has a verifier that checks the same success condition. Everyday Sep 2026 67.8% Not in cards
BU Bench V2.1 Browser agents do 200 real web tasks. A findings judge checks tool evidence, answers and files, and deterministic integrity checks run as well. Everyday Sep 2026 81.3 Not in cards
Internal computer use safety benchmark (OpenAI) Models complete computer-use tasks while avoiding unsafe actions. Research frontier Sep 2026 2.40% System cards only
LongPuzzleBench GUI agents solve 114 visual puzzle levels across six games through native interface actions. Everyday Sep 2026 Not in cards
MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. University Sep 2026 51.5% Not in cards
Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. Expert Sep 2026 1.40% System cards only
OSWorld 2.1 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. University Sep 2026 44.3% In use
OSWorld 2.1 (offline subset) Agents complete the 82 OSWorld 2.1 tasks on the official offline list, with no internet access. Scored by checkpoint partial credit and strict pass rate. University Sep 2026 89.3% In use
OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. Expert Sep 2026 73.7% Not in cards
PhoneShell iOS Agent Benchmark Agents operate apps on a physical iPhone to set timers, find books, and complete other everyday tasks. Automated checks read the resulting device state. Everyday Sep 2026 96.6% Not in cards
ProgramDistill Agents inspect working web applications and restore missing features in editable copies. Replayed browser traces check whether patches recover the reference behavior. The paper evaluates partial repairs and full application reconstruction. University Sep 2026 84.3% Not in cards
RecreationBench Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result. Expert Sep 2026 58.1% In use
SpeedrunBench Agents try to finish goals in 10 video games as fast as possible. Score is time to goal in frames or seconds, lower is better. Everyday Sep 2026 Not in cards
ADeptS-Bench Desktop Agents choose safe interface actions on the Desktop test set. The published safety score combines task success and attack resistance. Expert Aug 2026 76% Not in cards
CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. Expert Aug 2026 Not in cards
OSWorkerBench Desktop agents complete long-horizon office tasks across business applications with executable evaluators and optional demonstrations. Expert Aug 2026 Not in cards
OSWorld 2.0 (Aug 2026) Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. University Aug 2026 77.9% System cards only
Composite-Bench Agents solve generated browser tasks in planning, rule extraction, deduplication, routing, and information gathering. Success requires matching a certified optimal solution. A separate number-partitioning task measures solution quality rather than contributing to the success rate. Expert Jul 2026 74% Not in cards
ChainWorld Agents complete sequences of desktop tasks while preserving the state needed for later steps. OSWorld checks each step. Everyday Jun 2026 31% Not in cards
CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. Everyday Jun 2026 42.5% In use
Cua-Bench EDA (KiCad v1) Agents build or edit KiCad schematics through the desktop interface. The grader checks the resulting netlist and reports full passes, partial passes, and mean reward. Expert Jun 2026 24% Not in cards
DeskCraft Desktop agents complete 538 Ubuntu tasks, including 152 tasks with later user clarifications or interruptions. Programmatic checks inspect final application state, saved files, and exported artifacts. Expert Jun 2026 29.8% Not in cards
iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. Everyday Jun 2026 51.9% Not in cards
K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. Expert Jun 2026 45.7% Not in cards
MyPCBench Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks. Everyday Jun 2026 62% Not in cards
OSWorld 2.0 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. University Jun 2026 20.6% In use
PhoneHarness Bench Agents complete Android workflows through graphical controls, device commands, and structured tools. Checks inspect files, system settings, application state, and unintended side effects. Everyday Jun 2026 62.1% Not in cards
SentinelBench Agents monitor changing web environments and respond when scripted events make progress possible. Scores measure completion, reaction time and resource use. Expert Jun 2026 Not in cards
Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. University Jun 2026 60.2% Not in cards
WeaveBench Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts. Expert Jun 2026 41.2% Not in cards
ClawBench V2 Browser agents do 130 new everyday tasks on live websites under six supported harnesses. Request interception and a more lenient LLM judge score each run. Everyday May 2026 44.6% Not in cards
CutVerse Agents complete editing tasks in seven applications, including Premiere Pro, Photoshop, and DaVinci Resolve. Vision models grade progress against milestones in the recorded workflow. University May 2026 68.3% Not in cards
ERP-Bench Agents complete 300 procurement and manufacturing workflows in Odoo 19. A constraint solver generates the tasks and optimal solutions, and the verifier checks final business state. Expert May 2026 43.4% Not in cards
MobileGym Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state. Everyday May 2026 58.8% Not in cards
OpenComputer Computer-use agents complete 1,000 desktop tasks across 33 real applications, including browsers, office tools, creative software, development environments and messaging apps. App-specific verifiers inspect the final application state, and the harness gives partial credit from the recorded trajectory. Everyday May 2026 68.3% Not in cards
SaaS-Bench 106 long-horizon GUI workflows across 23 locally deployed real SaaS apps. Scored from app databases with weighted checkpoints and full-resolution rate. University May 2026 31.1% In use
UniGUI-Bench Agents interpret and operate graphical interfaces. The benchmark tests 11 capabilities and includes separate system evaluations. Everyday May 2026 80.9% Not in cards
WebHarbor Agents navigate local copies of 153 websites to search, sign in, book services, and change stored state. The suite contains 3,518 tasks, including 306 verified test tasks and 2,288 training tasks. The test score measures the percentage of tasks passed by scripted verifiers. Everyday May 2026 86.7% Not in cards
ClawBench Browser agents do 153 everyday tasks on live websites, such as booking travel, ordering food and applying for jobs. Request interception stops real side effects, and an LLM judge scores the five-layer traces. Everyday Apr 2026 61.4% Not in cards
CUA-World-Long Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task. Expert Apr 2026 27.5% Not in cards
GameWorld Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. Everyday Apr 2026 76.8% System cards only
KnowU-Bench Agents use Android apps such as Mattermost and Taodian. They infer preferences, ask for missing details, and decide when to act or stay silent. Everyday Apr 2026 60.4% Not in cards
GUI-CEval Models answer questions about Chinese app screens, locate controls, and choose actions. Agents also complete tasks on physical devices, where the grader checks interface elements. Everyday Mar 2026 61.4% Not in cards
AutoElicit An optimizer perturbs benign instructions to elicit unsafe behavior from computer-use agents. Reports elicitation success rate on transferred prompts. Expert Feb 2026 37.6% System cards only
MemGUI-Bench Agents remember details while working across Android apps. Tasks include recording audio and creating calendar events. A model checks the actions and results across repeated attempts. Everyday Feb 2026 73.4% Not in cards
Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. Expert Feb 2026 9.40% System cards only
VenusBench-Mobile Agents use Android apps such as VLC, Markor, and Pro Expense. A model judges 90 primary tasks. The other tasks use programmatic checks. Everyday Feb 2026 24.8% Not in cards
BU Bench V2 Browser agents do real web tasks with the Browser Use library. A judge scores each run against a rubric. Everyday Jan 2026 80.6 Not in cards
MobileWorld Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. Everyday Dec 2025 55.3% In use
WebArena-Verified Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. Everyday Dec 2025 69% In use
WorldTravel Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data. Everyday Dec 2025 32.7% Fading
AndroidDaily Agents complete 350 daily tasks in 94 real closed-source Android apps. A process-aware reviewer model checks each trajectory against written guidelines. Everyday Nov 2025 61.9% Fading
OSGym Infrastructure running 1,000+ parallel OS sandboxes for computer-use agent training and evaluation. Tasks span browsing, documents, SWE and multi-app work. Everyday Nov 2025 Not in cards
Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. Expert Nov 2025 0% System cards only
Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Expert Nov 2025 0% System cards only
WebTailBench Computer-use agents complete 609 hand-verified tasks on live websites across 11 categories, such as bookings, real estate and job search. A companion set of 111 harmful tasks tests refusals. Everyday Nov 2025 57.4% Fading
WebTailBench-Refusals Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused. Everyday Nov 2025 100% System cards only
OSWorld-MCP Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps. Everyday Oct 2025 56.6% Not in cards
WARC-Bench Screenshot-only agents do short web subtasks, such as date pickers and scroll containers, on real sites replayed from Web Archive snapshots. A program checks the end state. Everyday Oct 2025 64.8% Not in cards
Automotive-ENV Agents operate climate controls in an Android Automotive OS simulator. A validator checks the screen and vehicle signals after requests such as turning on the defroster. Everyday Sep 2025 49.3% Not in cards
DeepResearch Arena Deep research agents get research tasks generated from academic seminar transcripts, which reflect expert discussion and reduce data leakage. Expert Sep 2025 3.96 Not in cards
SCUBA Agents work in a Salesforce sandbox as administrators, sales representatives, or service agents. Evaluators check milestones such as creating accounts and routing support cases. University Sep 2025 53.9% Not in cards
UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. Everyday Sep 2025 59.8 Dropped
Android Agent Arena (100 tasks) Agents use 20 Android apps to find information and perform actions. A vision model checks whether the agent reached the required intermediate states and completed the task. Everyday Aug 2025 53% Not in cards
Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. Expert Aug 2025 99% System cards only
WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Everyday Aug 2025 83.9% In use
Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. Expert Jul 2025 67% System cards only
In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. Expert Jul 2025 78% System cards only
Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. Expert Jul 2025 95% System cards only
MMBench-GUI GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration. Everyday Jul 2025 81.5% System cards only
OSWorld-Verified 369 open-ended tasks in real desktop apps. Agent acts through screenshots, mouse and keyboard. Execution-based checkers score success. Everyday Jul 2025 86% In use
DeepShop Web agents answer complex shopping queries on live sites with product attributes, search filters and sorting preferences. An LLM judge checks whether results meet each requirement. Everyday Jun 2025 42.3% Fading
Mind2Web 2 Agents browse the live web for 130 long tasks and return answers with citations. Task-specific judge agents built from rubric trees grade correctness and attribution. Expert Jun 2025 28% Not in cards
ScreenSuite Models use screenshots to locate controls, select actions and complete computer tasks. The suite measures perception, single actions and complete interactions. Everyday Jun 2025 56% Not in cards
AndroidWorld-Knowledge Agents attempt 48 AndroidWorld tasks across five apps. Each app provides guidance for its simplest task, and harder tasks test whether the agent can reuse it. Everyday May 2025 Not in cards
Deep Research Bench (FutureSearch) Web research agents work on multi-step research tasks against RetroSearch, a frozen copy of 10,000 to 100,000 web pages per task. Answers are compared with answers worked out by people. Expert May 2025 0.55 Not in cards
RiOSWorld Computer-use agents work in an Ubuntu virtual machine through risky scenarios such as phishing emails and pop-ups. Scores track whether agents notice and avoid the risk. Everyday May 2025 79% Not in cards
ScienceBoard Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking. Expert May 2025 68% Not in cards
VideoGameBench Lite Agents play video games while the environment pauses during inference. The headline is the official Overall Score. Expert May 2025 1.60% Not in cards
WebBench (Halluminate) Browser agents complete 2,454 tasks on 452 live websites from the global top 1,000 by traffic. Tasks either read information or write data: creating, updating and deleting records, logging in and downloading files. The board compares browser agent products. Everyday May 2025 76.5% Not in cards
Online-Mind2Web Web agents do 300 tasks on 136 live websites. An LLM judge reads the agent's trajectory and decides whether the task succeeded. Everyday Apr 2025 69% Fading
REAL Agents complete 112 tasks on replicas of 11 websites. State checks grade actions, while a model judge checks retrieved answers against references. Everyday Apr 2025 41.1% Not in cards
AI Agent Privacy Web agents complete tasks while exposed to private information. The benchmark measures privacy leakage and unsafe disclosure during autonomous operation. Expert Mar 2025 Not in cards
Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. Everyday Feb 2025 99.4% System cards only
WorldGUI Desktop GUI agents do 611 tasks that start from varied initial states, such as the target app not being open. Tasks come with instructional videos and project files. Everyday Feb 2025 45.8% Not in cards
Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. Everyday Jan 2025 97% System cards only
Android Agent Arena (201 tasks) Agents find live information and perform actions in Android apps such as YouTube Music and Booking.com. Task functions check app state and grade answers. Everyday Jan 2025 10.3% Not in cards
Autonomy-10 Agents complete browser, desktop and tool-use tasks across nine domains that measure skill, flexibility and intelligence. Expert Jan 2025 Not in cards
Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. Everyday Jan 2025 Not in cards
Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. Expert Jan 2025 23% System cards only
Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. Everyday Jan 2025 94% System cards only
User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. Everyday Jan 2025 92% System cards only
TheAgentCompany Agents do work tasks in a simulated software company, using a browser, code, and chat with simulated coworkers. Scored by full and partial checkpoint completion. University Dec 2024 Dropped
AndroidLab Mobile agents complete 138 tasks across nine apps on predefined Android virtual devices, in text-only and multimodal modes. Checks on the final device state score each task. University Oct 2024 31.2% Not in cards
BrowserART Browser agents receive 100 harmful requests on synthetic and real websites. The test measures how often the agent attempts the harmful behavior that its backbone model would refuse in chat. Expert Oct 2024 11% Not in cards
SPA-Bench Agents use Android apps such as Airbnb, Spotify, and WeChat. The grader matches screenshots against task requirements and uses a vision model to resolve uncertain cases. Everyday Oct 2024 1.35% Not in cards
Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. Expert Sep 2024 System cards only
Windows Agent Arena Agents complete more than 150 tasks in a real Windows OS with its apps, tools and browsers. Scripts adapted from OSWorld check the final state. Everyday Sep 2024 50.6% Dropped
VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. Everyday Aug 2024 36.2% Not in cards
AssistantBench Web agents do 214 realistic, time-consuming tasks on the open web, such as monitoring real-estate markets. Answers are checked automatically against gold answers. Everyday Jul 2024 Not in cards
CRAB Multimodal agents do tasks that span an Ubuntu desktop and an Android phone at once. Graph-based evaluators give credit for intermediate steps. Everyday Jul 2024 38% Not in cards
Spider2-V Multimodal agents do data science and engineering workflows in real enterprise apps such as BigQuery, dbt and Airbyte, on a desktop computer. Execution-based checks score each task. Expert Jul 2024 14% Not in cards
WorkArena++ Web agents complete 682 compositional knowledge-work tasks in ServiceNow, which need planning, reasoning and memory across many steps. Expert Jul 2024 1.90% Not in cards
GUI Odyssey Mobile GUI agents carry out tasks that cross several apps, such as sharing a post from one social app to another. The dataset has 8,334 episodes averaging 15.3 steps, over 212 apps, 1,357 app combinations and six devices. Agents are scored by how often each predicted action matches the recorded step. Everyday Jun 2024 55% Not in cards
Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. Everyday Jun 2024 Not in cards
MobileAgentBench Agents complete 100 tasks across ten Android apps. The grader checks app state and accessibility events, so agents can take different paths. Everyday Jun 2024 8.00% Not in cards
WebCanvas Web agents do tasks on live websites, scored on key intermediate states rather than one fixed action path, so the test survives site changes. Everyday Jun 2024 28.8% Not in cards
AndroidWorld Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. Everyday May 2024 100% In use
MobileMiniWoB++ Agents complete MiniWoB++ web tasks inside the Android emulator of AndroidWorld. Scored by task success rate. Everyday May 2024 68% Dropped
B-MoCA Agents use Android apps such as Instagram on phones with different languages, font sizes, and icon positions. Task-specific checks decide whether each attempt succeeded. Everyday Apr 2024 47.7% Not in cards
LlamaTouch Agents carry out requests in Android apps. The evaluator checks whether each run visited the required app or system states. Everyday Apr 2024 Not in cards
OSWorld Agents complete 369 real computer tasks in Ubuntu desktop apps through screenshots and mouse and keyboard actions. Execution-based scripts check the final state. Everyday Apr 2024 78.8% Fading
WorkArena Web agents do routine knowledge-work tasks in a live ServiceNow instance, such as filling forms and filtering lists. WorkArena++ adds 682 compositional tasks at levels 2 and 3. University Mar 2024 Not in cards
AndroidArena Agents use apps such as Gmail and Google Maps, sometimes switching apps or following extra constraints. GPT-4 judges success from the recorded actions and observations. Everyday Feb 2024 12.7% Not in cards
OmniACT Models generate executable scripts from a screenshot and a user request. The evaluation compares the actions needed to complete desktop and web tasks. University Feb 2024 Not in cards
WebLINX Agents follow a user's multi-turn chat instructions to navigate real websites. 100,000 interactions from 2,300 expert demonstrations give turn-level metrics. Everyday Feb 2024 10.4 Not in cards
Mobile-Eval Agents follow 33 instructions on Android. Some requests specify the app. Others describe a goal and leave the choice of app to the agent. Everyday Jan 2024 Not in cards
VisualWebArena Multimodal web agents complete 910 visually grounded tasks on self-hosted classifieds, shopping and Reddit-style sites. Execution-based checks score each task. University Jan 2024 16.4% Not in cards
WebVoyager Agents complete user tasks on 15 popular real websites by browsing from screenshots and page text. A GPT-4V judge grades whether each task succeeded. Everyday Jan 2024 89.3% Fading
AppAgent Test Set Agents follow 45 instructions across nine Android apps, including scheduling email and changing music playback. Tasks time out after ten steps. Everyday Dec 2023 Not in cards
DroidTask Agents select actions for 158 Android tasks across 13 apps. The original completion score requires every action to match the recorded sequence. Everyday Aug 2023 Not in cards
WebArena Agents complete 812 realistic tasks on self-hosted shopping, forum, GitLab, CMS and map websites. Functional graders check the outcome. Everyday Jul 2023 68% Fading
WebShop Agents search and click through a simulated shopping site with 1.18 million real products to buy an item that matches a text instruction. Reward reflects how well the purchase matches. Everyday Jul 2022 Not in cards
MiniWoB++ Agents operate over 100 small synthetic web pages, such as forms, menus and date pickers, from a natural-language instruction. Each task returns a success reward. Everyday Feb 2018 Not in cards
AgentCLUE-Mobile Mobile assistants complete tasks on a phone. Everyday Not in cards
AgentCLUE-Mobile (offline) Agents select actions in an offline mobile interface. Everyday 88.4 Not in cards
BrowserInject v1 Browser agents do benign tasks on 30 retail, health, support and travel sites seeded with prompt injections. Reports attack success rate. Expert 36.7% System cards only
Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. Everyday 100% System cards only
Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. Everyday 100% System cards only
Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. Everyday 99.4% System cards only
Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. Expert 78% System cards only
Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. Expert 92% System cards only
OdysseysMega Agents complete tasks in a web browser. A judge checks each rubric criterion against the recorded actions and screenshots. Expert 41% Not in cards
Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. Expert 98% System cards only
SheetBench Agents solve finance and accounting spreadsheet tasks through a browser in a fresh Google Sheet. The score is the share of expected answer cells that match exactly. Expert 64.7% Not in cards
WebsiteBench Agents explore reference websites and rebuild their appearance, behavior, and persistent state. Hidden browser tests require functional correctness and screenshot similarity, then report the fraction passed. Expert 26.2% Not in cards