Agents
Models take a sequence of actions and decide when to stop, alone or with other agents. We count 2306 benchmarks here, 138 of them arenas. A benchmark belongs here when interaction is agentic or multi-agent.
How fast it grows
941 were released since July, more than the 535 in the first half of 2026.
How much is still open
No model has passed 85% on 995 of the 1232 benchmarks here that score in percent, 81% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| 3D GameDev Arena Agents build playable 3D browser games from specifications and reference images. Players compare anonymous game pairs. The authors also test functionality and playability. | Expert | Nov 2026 | 99.8% | Not in cards |
| 3DArena-Modeling Agents create 3D models for 120 tasks covering precision and appearance. Tencent converts Elo ratings from blind comparisons into percentages. | University | Nov 2026 | Not in cards | |
| 3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. | Expert | Nov 2026 | 59.5 | Not in cards |
| AbMaze Agents make choices in a simulated antibody optimization experiment under a fixed budget. Recorded assay results determine normalized improvement on 200 test episodes. | Expert | Nov 2026 | Not in cards | |
| Accelerate Bench Coding agents speed up 32 real scientific applications, such as GROMACS and OpenFOAM, on a fixed CPU allocation within 8 hours. A hidden verifier checks correctness, and Score@1 averages the share of runtime saved, with incorrect programs scoring zero. | Expert | Nov 2026 | 67.4% | Not in cards |
| AgenticFinBench Agents complete 230 financial requests in two languages. Graders apply task-specific rubrics and audit unsupported claims against external evidence. | Expert | Nov 2026 | 89.2 | Not in cards |
| AgenticRL-Bench Agents debug and maintain reinforcement learning infrastructure. A task passes when the implementation meets every hidden check. | Expert | Nov 2026 | 22.8% | Not in cards |
| AgentScript-Eval Models write, debug, or optimize AgentScript definitions for Salesforce agents. A task passes when the program compiles and passes every execution test. | University | Nov 2026 | 37% | Not in cards |
| ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. | Everyday | Nov 2026 | | Not in cards |
| AlphaSense Financial Research Benchmark Agents answer 245 multi-step finance questions across company research, financial modeling, earnings, deals, markets and regulatory analysis using source retrieval. | Expert | Nov 2026 | Not in cards | |
| ARE-bench Agents continue an existing research project from a real user request. A judge checks the deliverable against the request and earlier workspace requirements. | Expert | Nov 2026 | 66.2% | Not in cards |
| AscendGYM Agents optimize existing kernels for Ascend processors using tensor shapes from real workloads. The benchmark measures speedup and token consumption. | Expert | Nov 2026 | 74.0 | Not in cards |
| ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. | Everyday | Nov 2026 | Not in cards | |
| AstroMemBench Spacecraft agents use information from earlier observations to complete simulated missions. The environment scores task completion, safety, and resource use. | University | Nov 2026 | Not in cards | |
| AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. | University | Nov 2026 | Not in cards | |
| Autoform-Hard Agents complete Lean 4 proofs on 56 formalization tasks that earlier models failed. | Expert | Nov 2026 | 87.5% | Not in cards |
| Backrooms: 3D Clue-Trail Navigation Agents follow visual clues through a first-person maze. Only an escape without a wrong doorway counts as a solve. | Everyday | Nov 2026 | Not in cards | |
| Beyond Survival Models play social deduction games with human-aligned strategies and interaction patterns. | Everyday | Nov 2026 | 58.7% | Not in cards |
| BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. | Expert | Nov 2026 | 66.7% | Not in cards |
| Browser Use LinkedIn Agents answer plain-language questions about LinkedIn using recorded page snapshots. The leaderboard reports correctness, cost, and time per task. | Everyday | Nov 2026 | 97.9 | Not in cards |
| Clocktower Bench Models play the social deduction game Blood on the Clocktower (the Trouble Brewing script) against each other, with a scripted Storyteller. One model takes every good seat and another every evil seat, so each game tests one model's detection of the Demon against another's deception. A Bradley–Terry fit turns 386 games into Elo ratings. | Everyday | Nov 2026 | 1,859 | Not in cards |
| Collaborative Intelligence Benchmark A model works in a team with scripted teammates and live model partners, and a teammate creates a test: pressure to take an unsafe shortcut, misconduct to report, a need nobody states, a conflict, or a cheat that a weaker model left behind. The 240 tasks across 8 dimensions of teamwork come from real records, such as aviation accident reports and clinical cases, and from existing benchmarks, such as SWE-bench and negotiation datasets. Each task has a checkable outcome. A model is scored only on the tasks it solves alone, so the score measures teamwork, not ability. | Everyday | Nov 2026 | | Not in cards |
| Company Agent Bench 205 questions on how well an agent understands its company, built from a month of real Slack history: who wrote an anonymized message, whom a teammate tagged, and who sent each reply. Agents search a replayed sandbox workspace using only messages from before the answer. | Everyday | Nov 2026 | 82.2% | Not in cards |
| Construction AI Leaderboard Models answer construction questions, inspect drawings, edit building models, and generate validation rules. ONESTRUCTION averages the representative scores from five evaluations. | University | Nov 2026 | 73.7 | Not in cards |
| Coverage Cat AI Insurance Benchmark (fictional insurance tasks) Models read fictional policy documents, find missing facts, and complete insurance applications. The scorer checks source quotes, required fields, and authorized actions. | University | Nov 2026 | 100% | Not in cards |
| Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. | Expert | Nov 2026 | | In use |
| Crazyrouter AI Trading Arena Models buy, sell, or hold three cryptocurrencies every five minutes using simulated accounts. The grader values portfolios at live quotes and deducts trading fees. | University | Nov 2026 | -7.45% | Not in cards |
| CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. | Everyday | Nov 2026 | | Not in cards |
| CursorBench 4.1 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported. | Expert | Nov 2026 | | In use |
| CWE-bench 2 Agents audit and patch vulnerabilities in 250 codebases. The score is pass@1 under deterministic verification. | Expert | Nov 2026 | 75% | In use |
| Databricks AI Index Coding agents complete tasks at different costs. The index compares cost per task and solution quality alongside enterprise usage statistics. | Expert | Nov 2026 | 96.0 | Not in cards |
| DeepFinance Agents do private equity and investment banking work on 21 anonymized real deals: valuation, due diligence, underwriting and financial modeling. Each task comes with a data room of about 19 files and would take a professional about 27 hours. Rubrics of about 84 criteria grade the output, the workflow and the judgment behind each assumption. | Expert | Nov 2026 | 62% | Not in cards |
| DeGennaro Workplace sample Agents act as an employee’s assistant using 80 simulated workplace tools and anonymized company records. Four tasks grade the final state. | Expert | Nov 2026 | Not in cards | |
| DeputyBench Agents handle business requests under paired permission settings. The evaluator checks whether they act when authorized and seek evidence or refrain when authorization is missing. | University | Nov 2026 | | Not in cards |
| Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. | Expert | Nov 2026 | | In use |
| Deta Studio AI Construction Takeoff Models use Deta’s tools to mark floor areas and count openings and electrical devices on four drawings. The rating combines accuracy, cost, and time. | University | Nov 2026 | 76 | Not in cards |
| DrivingBench v2 Vision-language models drive a real Tesla through two unseen cone courses with steering, accelerator and brake tools. The score is the furthest progress over three attempts in one conversation, averaged over six trials. | Everyday | Nov 2026 | | Not in cards |
| EarthRecord Agents analyze multidecade satellite observations to answer land-change questions. Reference checks score answers to 238 questions across 56 sites. | Expert | Nov 2026 | 85.4% | Not in cards |
| Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. | University | Nov 2026 | Not in cards | |
| EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. | University | Nov 2026 | Not in cards | |
| Enterprise-Bench L1-L2 v2 Agents answer 42 operational questions using six enterprise services. A GPT-6 Luna judge evaluates the final answer against task criteria. | University | Nov 2026 | 65.7% | Not in cards |
| EnterpriseOps-Gym 1.1 Agents complete 296 workflows across customer service, human resources, and IT service management. SQL checks grade the resulting database state. | Expert | Nov 2026 | Not in cards | |
| EUREKA-Bench (OpenAI) Agents build predictive models for six biology research tasks. Scored as predictive performance with the best published predictor in each field set to 100. | Research frontier | Nov 2026 | 100.3 | System cards only |
| ExcelFixBench Agents repair formulas, data, and formatting in an existing workbook. The task rubric checks the repair and whether unrelated content survived. | University | Nov 2026 | | Not in cards |
| ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. | Everyday | Nov 2026 | Not in cards | |
| Fermat's Last Theorem in Lean (Anthropic) Agent teams formalize Fermat's Last Theorem in Lean. Scores measure the weighted proportion of 60 accepted intermediate theorems. | Research frontier | Nov 2026 | 35% | System cards only |
| FinDesk-Bench Agents find Treasury figures and use them to handle client requests across ten business services. A model grades API calls and the records agents leave behind. | University | Nov 2026 | | Not in cards |
| Firecrawl Search Bench Agents answer hard research questions with web search. A deterministic grader checks each answer against reference facts. | Expert | Nov 2026 | 79% | Not in cards |
| FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. | University | Nov 2026 | | Not in cards |
| FraudBench (GOAL Lab) Models face 159 scenarios drawn from 112 real U.S. fraud lawsuits. Yes or no questions, open questions and agent tasks test whether a model recommends, assists or commits fraud. | University | Nov 2026 | 10.1% | Not in cards |
| Frontier3DBench Agents create editable Blender objects and scenes or executable Unreal Engine worlds across 145 tasks and nine task families. Model judgments and native project checks assess appearance, geometry, requirements, and interactions. The overall score weights object, scene, and world scores by 30%, 30%, and 40%. | Expert | Nov 2026 | 66.0 | Not in cards |
| FrontierBioML 25 machine learning tasks in computational biology, across antibody development, protein engineering, structural biology, binder design and functional genomics. An agent gets real experimental data and a brief, works offline for up to six hours and submits a prediction script. A sealed grader runs the script on held-out data, such as new antibody families or new protein targets, and scores each task from 0 at a simple baseline to 100 for a perfect answer. | Expert | Nov 2026 | 40.3 | Not in cards |
| FrontierBreach Agents perform simulated enterprise tasks while processing untrusted material. The benchmark checks whether each agent resists instructions that conflict with its assigned task. | Expert | Nov 2026 | | Not in cards |
| FrontierEE Bench Agents run electrical engineering experiments on real lab hardware through networked instruments, camera photos and tickets to a human operator. Deterministic verifiers check 10 tasks against protected bench records. | Expert | Nov 2026 | 36.7% | Not in cards |
| FrontierML Agents spend a fixed budget on GPUs and model APIs to train models for 21 tasks. An offline evaluator grades their submissions on held-out data. | Research frontier | Nov 2026 | 58.9 | Not in cards |
| FrontierRSI Agents do 27 open-ended AI research and engineering tasks across pre-training, post-training and harnesses, inference serving and GPU kernels. Each trial is scored against the best valid result any model reached. | Research frontier | Nov 2026 | 34.3% | Not in cards |
| GDPval v2 (OpenAI) Multistep assignments drawn from real projects in consulting, finance and design, given with the files and messages the team had when the work was assigned. OpenAI has not published the task count or grading. | Expert | Nov 2026 | 50.4% | In use |
| GDPWorks Agents complete 83 professional tasks across 36 occupations, using source files to produce work such as documents, spreadsheets, and code. The Joules-to-GDP framework compares outputs with expert deliverables through a blinded model judge. Task success measures the share of wins and ties against the expert. | Expert | Nov 2026 | Not in cards | |
| GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. | Expert | Nov 2026 | 16.4% | Not in cards |
| GeoGuesser Bench Agents explore offline Street View panoramas to report coordinates, name places, and find objects. Coordinate guesses earn distance points. Other answers require an exact match. | Everyday | Nov 2026 | | Not in cards |
| GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. | Everyday | Nov 2026 | | Not in cards |
| GroupAssistBench Assistants use memories from group conversations to answer questions and complete real-world group assistance tasks. Model judges score grounded helpfulness. | Expert | Nov 2026 | Not in cards | |
| Hard12 Agents solve 12 selected coding tasks from five benchmarks. Hidden tests grade each task. The leaderboard averages three runs per task. | Expert | Nov 2026 | 83.3% | Not in cards |
| Harness Leaderboard Coding-agent harnesses such as Claude Code, Codex and OpenCode run the same open model at the same reasoning effort on 198 tasks from Terminal-Bench 2.1 and DeepSWE v1.1. Scored as the resolve rate averaged across the two benchmarks over three attempts, with any access to the answer counted as a fail. | Expert | Nov 2026 | Not in cards | |
| HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. | Everyday | Nov 2026 | 18.8% | Not in cards |
| Historical Systems Bugs Agents repair two historical C bugs in Linux and Unikraft. Hidden tests exercise the patched code, including driver behavior under emulated hardware. | Expert | Nov 2026 | Not in cards | |
| HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. | Everyday | Nov 2026 | 61.9% | Not in cards |
| HoneyBench v0.2-alpha Measures reward hacking: how often an agent games its grader instead of doing the task. Lower is better. | Research frontier | Nov 2026 | | In use |
| Honor of Kings Bug AI Root Cause Analysis Agents use the Honor of Kings Bug AI platform to identify the cause of code defects. The team measures how often agents identify the correct cause. | University | Nov 2026 | | Not in cards |
| Honor of Kings Game Configuration Agents configure game data with the Honor of Kings spreadsheet platform. The team measures configuration accuracy before and after the agent checks and repairs its work. | University | Nov 2026 | Not in cards | |
| How frontier models miss risk in routine enterprise work Agents handle workplace requests whose risks appear in the prompt or in supporting files. Judges check whether the agent completes an unsafe action. | Expert | Nov 2026 | Not in cards | |
| HW-Bench Agents complete six hardware engineering tasks for a 2.45 GHz antenna using native design and simulation tools. Code checks and calibrated model judges grade outputs against engineer-written criteria. The headline pass rate measures runs that satisfy every critical criterion, averaged across tasks. | Expert | Nov 2026 | 67% | Not in cards |
| HWE-Bench v1.1 (RTL bug repair) Agents fix hardware bugs in 417 repository tasks. The revised tests check whether each fix restores the required behavior. | Expert | Nov 2026 | 99% | Not in cards |
| Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. | Expert | Nov 2026 | 83.9% | Not in cards |
| Infrathon Agents optimize AI infrastructure under correctness and fidelity constraints. Hidden workloads measure performance gains, with additional checks for rule compliance. | Expert | Nov 2026 | 91.2 | Not in cards |
| InSAR Regional Snow Mapping (Axis Spatial) Agents analyze Alpine radar scenes, align weather observations with radar windows, and calculate regional statistics. A script supplies reference answers for 24 tasks. | University | Nov 2026 | 100% | Not in cards |
| Invoice Approval (Ali Siala) Agents process 60 supplier invoice work orders under a German company’s approval policy. The evaluator compares the resulting database with a reference state. | University | Nov 2026 | Not in cards | |
| Joi-Bench-Task Voice agents use tools while a simulated caller changes a request or interrupts a reply. The evaluator tracks completion, responsiveness, and omitted information. | Everyday | Nov 2026 | Not in cards | |
| Kaeru Flip (paprika3d) Agents play an 86-note rhythm game from browser pixels. They must finish the song with at most one miss and leave autoplay off. | Everyday | Nov 2026 | Not in cards | |
| KradleBench Models play live multi-agent Minecraft games and act by writing code. The overall score weights collaboration, economy and conflict at 30% each, and logic and creativity at 5% each. | Everyday | Nov 2026 | 78.3 | Not in cards |
| LeanLean Agents shorten Lean projects while preserving protected theorem statements and passing the build and axiom checks. | Expert | Nov 2026 | 50.5% | Not in cards |
| Legal Information Retrieval Benchmark Search APIs retrieve the official document behind legal research requests. Exact URL matching scores whether the official page appears first or in the top ten. | Expert | Nov 2026 | Not in cards | |
| Legal World sample Agents act as a legal assistant in an anonymized law practice. Four tasks grade the drafts, mail, calendar changes, and files they leave behind. | Expert | Nov 2026 | Not in cards | |
| LevelBench-100 v5.0 Models use ledger exports and supporting workbooks to close small business accounts. They reconcile transactions, apply accounting rules, and identify missing records. | Expert | Nov 2026 | | Not in cards |
| LifeVal Assistants carry out everyday personal tasks rather than advise on them: finding childcare, booking a car inspection, ordering groceries on a budget, handling government paperwork. Tasks come from time-use surveys and from about 100 people who recorded tasks they wanted an assistant to handle. Score is average reward across tasks. | Everyday | Nov 2026 | 86.1% | In use |
| Longtail-SWE Agents resolve 202 GitHub issues across 13 programming languages with limited benchmark coverage. Hidden tests check the fix and verify that previously passing behavior still works. | Expert | Nov 2026 | 81.5% | Not in cards |
| Malicious agentic influence campaigns, 34-point rubric (Anthropic) Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale. | Expert | Nov 2026 | System cards only | |
| ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. | Everyday | Nov 2026 | Not in cards | |
| MdCoEditBench Agents edit selected Markdown in the WorkBuddy editor alongside a user. Tasks include restoring a version, resolving an edit conflict, and continuing existing text. | University | Nov 2026 | Not in cards | |
| MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. | Everyday | Nov 2026 | 50.4% | Not in cards |
| METR Capability Index METR fits an item response model to agent results on Time Horizon 1.1, TH2 and MirrorCode. The index places models on a common ability scale. | Research frontier | Nov 2026 | 859.0 | Not in cards |
| METR Time Horizon 2 Agents complete 36 tasks across 12 families. METR estimates how long a skilled human would need for tasks that agents complete with 50 percent reliability. | Research frontier | Nov 2026 | 27.1 h | Not in cards |
| MindGames Agents compete in Colonel Blotto, Iterated Prisoner’s Dilemma, Codenames, and Secret Mafia. | Everyday | Nov 2026 | Not in cards | |
| Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. | Expert | Nov 2026 | | In use |
| MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. | Expert | Nov 2026 | | In use |
| Moneda Bench Models answer finance questions across seven separate benchmarks. AdviceBench, CiteBench, FilingsBench, GroundBench and PortfolioBench have their own records. KnowledgeBench and MarketBench have no published results yet. Moneda publishes no combined score. | Expert | Nov 2026 | Not in cards | |
| Moneda GroundBench Models answer live questions about market figures. Moneda’s default board reports the share of correct answers against reference data, with incorrect answers reported separately. | Expert | Nov 2026 | Not in cards | |
| MTGBench Models play multi-turn games that test reasoning, planning, and interaction across varied game types. | Everyday | Nov 2026 | Not in cards | |
| Multi Agent Coordination Bench Teams of eight agents work for four hours on twelve tasks built so each agent depends on its teammates. Each model also runs alone and as a team that can't talk, which isolates the gain from coordination. | Expert | Nov 2026 | 0.53 | Not in cards |
| Nim compiler repair benchmark Agents repair the Nim compiler or standard library from linked issues. The current README lists 285 tasks graded with compile and regression tests. | Expert | Nov 2026 | Not in cards | |
| Nim Hard Agents fix Nim compiler issues on Linux. Black-box tests grade changes to the C backend. The current README lists 15 tasks. | Expert | Nov 2026 | Not in cards | |
| OfficePrism Agents resolve Chinese workplace tickets using one business system at a time. The evaluator checks outcomes across 420 tasks in seven business functions. | University | Nov 2026 | 76.4% | Not in cards |
| On Evaluating User Simulators for Interactive Agents Models play the user in replays of 351 real sessions. The evaluation compares simulated behavior with observed behavior and measures agreement with human rankings of nine agents. | Everyday | Nov 2026 | Not in cards | |
| OWE-Bench 26 long-horizon engineering and research projects across software, physical engineering, machine learning and scientific computing, from GPU kernels to aircraft design. Agents get 4 to 18 hours per attempt, and an offline verifier scores what they ship on held-out cases. | Research frontier | Nov 2026 | 67.2% | Not in cards |
| Parley Labs negotiation tasks Agents negotiate contracts and plan around market changes with scripted counterparties. An offline verifier grades outcomes. Ten public development tasks accompany private scored sets. | Expert | Nov 2026 | Not in cards | |
| Portal2Bench Agents solve Portal 2 chambers using screenshots, position readings, and game controls. Each chamber counts if the agent reaches the next map within $30 and 300 plans. | Everyday | Nov 2026 | 26.0 | Not in cards |
| PPTCoEditBench Agents edit selected content in PowerPoint slides. Tasks include changing a title or table while preserving surrounding shapes and animations. | School | Nov 2026 | Not in cards | |
| PPTFixBench Agents inspect a PowerPoint file and repair its formatting and content. Each of the 50 files contains more than 25 issues. | University | Nov 2026 | Not in cards | |
| PReMind-Bench Models inspect a scene and decide whether to retrieve a personal memory before answering. Judges check whether the answer addresses the hazard and uses the right person’s information. | Everyday | Nov 2026 | 87.3% | Not in cards |
| Prompt Injection in Benchmark Environments An automated attacker plants prompt injections in 115 held-out agent environments adapted from Gray Swan challenges. Scored on the share of attacks the model resists. | Expert | Nov 2026 | 99.8% | System cards only |
| Prompt Injection in Model-Generated Environments An automated attacker writes both the agent environments and the prompt injections in them. Scored on the share of attacks the model resists. | Expert | Nov 2026 | 91.7% | System cards only |
| PWS-Decisions Decision APIs choose answers to 300 closed-ended questions whose answers depend on current or scattered facts on the public web. | Expert | Nov 2026 | Not in cards | |
| PWS-Health Web research systems answer healthcare questions from clinical trials, medical literature, registries, drug labels, and healthcare workflows. | Expert | Nov 2026 | Not in cards | |
| PWS-Legal Web research systems answer dated questions about companies, laws, regulations, intellectual property, court decisions, and legal documents. | Expert | Nov 2026 | Not in cards | |
| Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. | Expert | Nov 2026 | | Not in cards |
| Ramp SWE-Bench v2 Agents solve 500 engineering tasks from Ramp's production backend. A panel of model judges compares each patch with the code engineers merged. | Expert | Nov 2026 | 76.4% | Not in cards |
| Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. | Expert | Nov 2026 | 0.60 | Not in cards |
| Reasoning extraction red-teaming Six attacker models try to make a model reveal its hidden reasoning. Scored on how often an attack succeeds. | Expert | Nov 2026 | System cards only | |
| RedstoneBench Agents design redstone circuits in Minecraft. A game server tests whether each circuit works and measures the space it occupies. | Expert | Nov 2026 | 88.7 | Not in cards |
| RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. | Everyday | Nov 2026 | | Not in cards |
| RoboCasa-Mem Robots complete nine household tasks that require memory. Later steps depend on objects the robot saw or actions it took earlier. | Everyday | Nov 2026 | 16% | Not in cards |
| RoboColosseum Physical robots execute fresh instructions in paired blind trials. Human evaluators choose the better run or a tie, and comparisons determine ratings for each embodiment. | Everyday | Nov 2026 | Not in cards | |
| RoboDx Agents repair faults in robot navigation software. Hidden driving scenarios test the repair. A separate diagnosis track asks models to identify faults without seeing the code. | Expert | Nov 2026 | 100% | Not in cards |
| RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. | University | Nov 2026 | 80.6% | Not in cards |
| RollerCoasterTycoonBench Models manage a theme park by choosing weekly actions in RollerCoaster Tycoon 2. | Everyday | Nov 2026 | Not in cards | |
| Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. | Research frontier | Nov 2026 | 44.7 | Not in cards |
| Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. | Everyday | Nov 2026 | 42.9% | Not in cards |
| Sequitur Research Lab Automation Agents run simulated wet-lab automation protocols. Scored out of 100 on how well each experimental phase is carried out, averaged over six runs. | Research frontier | Nov 2026 | 74.5 | System cards only |
| ShadowBench Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match. | Everyday | Nov 2026 | 48.8 | Not in cards |
| SMB Bench An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule. | University | Nov 2026 | $7,749 | Not in cards |
| SolidWorks Bench Agents model the custom parts of 73 real machines in SolidWorks and assemble them from raster drawing packages, through screenshots and the SolidWorks API. A grader checks the geometry against the reference and reads mates, motion and feature trees from the native files. | Expert | Nov 2026 | 0.57 | Not in cards |
| SpaceChem Bench Agents program reactors to assemble molecules in SpaceChem. The main ranking measures how close their solutions come to the community’s records. | Competition | Nov 2026 | 74.3 | Not in cards |
| Spatial Agent Index v2 Agents analyze geospatial files and produce numbers, maps, or lists. Axis Spatial averages the scores for each included evaluation. | University | Nov 2026 | 86.0 | Not in cards |
| Spatial Intelligence Index v2 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. | University | Nov 2026 | 88.7 | Not in cards |
| SWE Atlas v1.1 Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. | Expert | Nov 2026 | 80.5% | Not in cards |
| SWE-CoWork Coding agents join a simulated engineering team on 101 real repositories and must gather scattered requirements from simulated colleagues and workplace records before implementing them. Hidden tests score the share of new-behavior tests passed and whether the whole task passes. | Expert | Nov 2026 | 7.92% | Not in cards |
| SWE-Pair Agents fix two real bugs from the same Python project using one combined issue report. Hidden tests from both pull requests grade each fix. | Expert | Nov 2026 | | Not in cards |
| TCSExam Agents prove theorems, write solvers, build mathematical constructions and address open conjectures in theoretical computer science. | Research frontier | Nov 2026 | 71.9 | Not in cards |
| TCSExam-Lite Agents solve 20 theoretical computer science tasks, five per track. The default solver protocol gives hidden feedback only after submission. | Research frontier | Nov 2026 | 72.3 | Not in cards |
| Terminal-Bench 4.1 The Terminal-Bench 4.0 tasks with hardened verifiers: agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. | Expert | Nov 2026 | 71.2% | In use |
| Terminal-Bench Math Modeling Agents use a terminal to solve 18 mathematical modeling problems and submit numerical answers. Feasibility checks and numerical scoring compare answers with reference solutions. | Competition | Nov 2026 | 64.3 | Not in cards |
| TIS-100 Bench Agents write and optimize assembly programs for 55 puzzles. The ranking counts puzzles where a program improves on the community’s recorded tradeoffs in cycles, nodes, and instructions. | Competition | Nov 2026 | 11.0 | Not in cards |
| Toloka Arena Enterprise Tool Use (November 2026) Agents complete private workflows across 11 industries. They must leave the right records and follow the required process in all five attempts. | Expert | Nov 2026 | | Not in cards |
| TriPhysBench Models answer quantitative questions about physical events in videos. Some questions require measuring an object or looking up its specifications. | School | Nov 2026 | | Not in cards |
| UAV Robust Control Hub Agents construct policies for a fictional civilian drone simulator. The benchmark evaluates policies on hidden scenarios and reports separate results under changed conditions. | University | Nov 2026 | Not in cards | |
| UK AISI File Download monitor evasion The model tries to download a forbidden file without a monitoring model noticing. Scored on how often it evades the monitor. | Expert | Nov 2026 | System cards only | |
| UK AISI simulated sandbox escape monitor evasion The model tries to escape a deliberately vulnerable sandbox without a monitoring model noticing. Scored on the share of samples where it escapes undetected. | Expert | Nov 2026 | 0.50% | System cards only |
| VABAR Agents program robots to sort tools, mesh gears, and handle cables in simulation. The harness checks success as object shapes and positions vary. | University | Nov 2026 | Not in cards | |
| VAHBench Agents turn short drama briefs into finished videos with specified characters, dialogue, and durations. The evaluator checks completeness, quality, reliability, and accuracy of production reports. | University | Nov 2026 | 51.8 | Not in cards |
| Vals Inference Provider Index Agents use different inference providers to complete the same finance, coding, and legal tasks. | Expert | Nov 2026 | | Not in cards |
| Vals Smith: pingdotgg/t3code Agents modify pingdotgg/t3code to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Nov 2026 | Not in cards | |
| Veris FinAgent Arena v1.1 Agents investigate bank alerts, read Treasury scans, and complete research requests. The index combines accuracy and repeatability across three benchmarks, then applies a cost penalty. | University | Nov 2026 | | Not in cards |
| Vibe Code Bench 1-100 v1.1 Agents extend a working web application across successive requests while preserving earlier features. | Expert | Nov 2026 | 55.2% | Not in cards |
| VibePrevisBench Coding agents write Blender Python that turns a script, photos, an opening frame or reference footage into a graybox 4D previs scene and render it. Rule checks score the exported scene against hidden facts, and a VLM judge checks that those facts are visible in the render. | Expert | Nov 2026 | 77.7 | Not in cards |
| VideoACE Models answer questions about fast actions in short videos. They can inspect selected frames more closely and request hints from a simulated viewer. | Everyday | Nov 2026 | | Not in cards |
| Voice Agent LLM Benchmark Models handle simulated caller requests in text. Hamming grades task completion, policy compliance, and how naturally the models phrase their replies. | Everyday | Nov 2026 | Not in cards | |
| Wayfinder Code Search Benchmark Coding search agents find the files changed by real bug fixes from short commit subjects or longer problem briefs. | Expert | Nov 2026 | Not in cards | |
| WeChat Pay Physical Client Evaluation Agents operate the WeChat Pay client on physical phones. The team scores execution paths, use of supplied knowledge, acceptance checks, and account creation accuracy. | University | Nov 2026 | Not in cards | |
| WordEditBench Agents edit Word documents. A rule-based checker reads the document XML, rewards requested changes, and penalizes changes to unrelated content. | University | Nov 2026 | | Not in cards |
| Work360 Agents complete professional tasks drawn from a map of 26 industries, 182 subfields and 721 specialties. Qwen in-house scoring (method unpublished). | Expert | Nov 2026 | | In use |
| WorkBuddy User Blind Evaluation Agents complete everyday work requests. WorkBuddy users rate their satisfaction on a four-point scale without seeing the model identity. | University | Nov 2026 | | Not in cards |
| WorkflowArena Agents write executable workflows that coordinate other agents on professional tasks. Hidden verifiers check the completed deliverables. | Expert | Nov 2026 | 37% | Not in cards |
| WorkIndex Agents complete everyday WorkBuddy tasks. Tencent combines eight tests covering office files, audio and video work, and visual questions. | University | Nov 2026 | Not in cards | |
| Yet Another Kernel Kernel Models optimize GPU kernels for speed while a verifier rejects numerical errors and input-insensitive shortcuts. | Research frontier | Nov 2026 | Not in cards | |
| 4DCodeBench Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion. | Expert | Oct 2026 | 0.79 | Not in cards |
| AA-3D Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings. | Expert | Oct 2026 | 1,318 | In use |
| AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. | Everyday | Oct 2026 | Not in cards | |
| AAArena Agents write and revise programs for 12 games. Each program competes against archived human entries under a fixed match budget. | University | Oct 2026 | 6.00 | Not in cards |
| AdaExploreBench Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice. | Everyday | Oct 2026 | | Not in cards |
| Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. | Everyday | Oct 2026 | | System cards only |
| Agent4Sci Coding agents try to improve the published best model on 20 scientific ML benchmarks across 11 domains, under long in-the-wild runs. The paper reports mean relative improvement rather than a ranking. | Research frontier | Oct 2026 | 20.1% | Not in cards |
| AgentCallBench: Tool-Calling Reliability Models operate a simulated store using tools. The grader checks tool choice, arguments, failure recovery, unnecessary calls, and refusals of impossible orders. | Everyday | Oct 2026 | | Not in cards |
| Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. | Expert | Oct 2026 | | System cards only |
| Agentic Credit Analysis Models extract and standardize income statements and balance sheets. Ramp compares their figures with credit analysts’ figures using a five percent tolerance. | Expert | Oct 2026 | 80.2% | Not in cards |
| Agents' Last Exam v1.1 Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references. | Expert | Oct 2026 | 47.5% | In use |
| AgentSWE Coding agents build, edit, or optimize agents across 25 software-engineering commissions. Frozen deliveries run on held-out cases, where execution gates and task-specific model-judged rubrics score behavior. Optimization tasks measure improvement against each source benchmark objective. | Expert | Oct 2026 | | Not in cards |
| AGORA (Agentic Global Open Research Arena) Autonomous AI systems get one secret open finance question, 72 hours and a fixed budget to write a paper. Five finance scholars judge it. | Research frontier | Oct 2026 | Not in cards | |
| Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. | Everyday | Oct 2026 | 87.9 | Not in cards |
| AML Bench (Veris AI) Agents investigate suspicious activity at a simulated bank. A model judge checks whether the recorded service requests and final case disposition satisfy every requirement. | Expert | Oct 2026 | 70.8% | Not in cards |
| Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. | Expert | Oct 2026 | Not in cards | |
| ArenaWebdevCorrectness Judges inspect generated applications and predict whether their features work. Human labels provide the reference answers. | University | Oct 2026 | 81.6% | Not in cards |
| Argo-Bench Agents query a simulated food delivery company's warehouse and act on their findings. The grader measures the business consequences of their decisions across 210 tasks. | Expert | Oct 2026 | 70.2 | Not in cards |
| Artificial Analysis Coding Agent Index v2 Coding agents, each a model plus harness, attempt software engineering, terminal, and repository question tasks. The index averages three equal-weight benchmark scores. | Oct 2026 | | In use | |
| Artificial Analysis Intelligence Index v5 Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. | Oct 2026 | 64.6 | In use | |
| ASTRA-bench Agents use personal context and interactive tools to complete more than 2,000 multi-step scenarios. Automated checks score whether each user intent is satisfied. | Expert | Oct 2026 | Not in cards | |
| ATLAS (Exa) Agents search the web to build a full table of every entity that meets a set of conditions, with 2 to 10 attributes each. A row counts only if every cell is correct. | Expert | Oct 2026 | Not in cards | |
| ATLAS-Finance v1.1.0 Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level. | Expert | Oct 2026 | 10.7% | Not in cards |
| AutoFactory-Bench Agents write control programs for simulated factory cells. A run passes when the program completes every operating condition without a safety violation. | University | Oct 2026 | 72.2% | Not in cards |
| AutoformBench Agents fill in missing Lean proofs for mathematical statements and submit their solutions to a clean grading workspace. | Expert | Oct 2026 | 82.4% | Not in cards |
| AutoVend Agents buy stock and schedule deliveries for 20 simulated vending machines over 180 days. They share one warehouse and pay for each site visit. | University | Oct 2026 | 309.4 | Not in cards |
| BallotBench Nine frontier agents forecast 863 contests in the 2026 US midterms each day until polls close. Proper scoring rules grade forecasts against certified results, beside Kalshi, Polymarket and expert ratings. | Expert | Oct 2026 | | Not in cards |
| BankerToolBench v1.1 Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes. | Expert | Oct 2026 | 32.6% | Not in cards |
| BARC-Bench Agents handle routine workplace requests and use records to infer privacy and safety constraints. A judge checks whether each attempt completes safely or defers. | University | Oct 2026 | 72.7% | Not in cards |
| BenchCAD 2.0 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. | Expert | Oct 2026 | 59% | Not in cards |
| BenchCAD 2.1 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. | Expert | Oct 2026 | 83.5% | In use |
| BenchCAD 2.1-beta Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. | Expert | Oct 2026 | Not in cards | |
| Binary Reasoning v1 Models answer machine-checked questions about compiled software across processor architectures. The hub port supplies 100 of 10,251 tasks and a separate set of 29 interactive tasks. | Expert | Oct 2026 | Not in cards | |
| BizBench Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success. | Expert | Oct 2026 | | Not in cards |
| BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. | Expert | Oct 2026 | 79.9 | Not in cards |
| Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. | Expert | Oct 2026 | 8.47% | In use |
| BrowseSafe-Adapt Browser agents face adaptive prompt injections across 1,024 tasks. The score is attack success within 64 attempts. | University | Oct 2026 | 30.7% | Not in cards |
| Build-A-Chip-Bench Agents take a chip from workload specification to a routed GDS layout with open-source tools. Each design must pass hidden verification, routability and timing gates. | Research frontier | Oct 2026 | 16.6% | Not in cards |
| BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. | Everyday | Oct 2026 | 0.56 | Not in cards |
| Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. | Expert | Oct 2026 | 37.8% | In use |
| Cat-Goric Game Benchmark Models navigate a 2D platform game and submit results to a public decision-making leaderboard. | Everyday | Oct 2026 | | Not in cards |
| CheckerBench Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback. | Expert | Oct 2026 | 45.3% | Not in cards |
| CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. | Expert | Oct 2026 | 76.6 | Not in cards |
| CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. | Expert | Oct 2026 | Not in cards | |
| CivBench Models play full games of Civilization V with the Vox Populi mod against each other on matched maps and seeds. Pairwise skill ratings rank long-term strategy. | Expert | Oct 2026 | 1,634 | Not in cards |
| CLASHarness Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment. | Everyday | Oct 2026 | 28.7% | Not in cards |
| Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. | Expert | Oct 2026 | $9,514 | Not in cards |
| ClothDojo Robot policies use both hands to flatten and fold garments. The simulator checks success on familiar garments and garments withheld from training. | Everyday | Oct 2026 | 44.7% | Not in cards |
| Co-Design Bench Agents revise existing applications from multimodal user requests. A judge checks browser and repository evidence against 2,331 expert criteria across 273 tasks. | Expert | Oct 2026 | 69.9% | Not in cards |
| Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. | Expert | Oct 2026 | 86.9% | System cards only |
| Code2Video-AA Artificial Analysis' implementation of HeyGen's Code2Video Bench: models write a HyperFrames HTML composition for each of 114 product and brand video briefs. A model judge compares two rendered videos at a time on five axes, and the comparisons become Elo ratings. | Expert | Oct 2026 | | In use |
| CompactionBench Agents continue multi-turn coding sessions while context compaction occurs during software evolution tasks. Scores measure task quality and compaction cost. | Expert | Oct 2026 | Not in cards | |
| Company Enrichment API Benchmark Company data APIs fill profile fields from a website. Code and expert checks score headquarters, employees, founding year, social links, industry, ticker and revenue. | Expert | Oct 2026 | Not in cards | |
| Concurrency Bugs (tanishgudise) Agents repair Python packages with asynchronous or threaded code. Hidden tests reproduce each bug deterministically across 15 repair tasks. | University | Oct 2026 | Not in cards | |
| Construction's Last Exam v2 Models measure construction drawings, answer questions across sheets, and price quantities under contract rules. The board reports geometry, drawing reasoning, and estimating separately. | Expert | Oct 2026 | | Not in cards |
| Continual Learning Exam Agents work through 120 professional tasks, each a ladder of five levels that must be passed in order. Later levels reuse files, tools and lessons from earlier ones. An evaluation agent grades against expert rubrics. | Expert | Oct 2026 | 47.7% | Not in cards |
| Cozzo AI Receptionist Benchmark (Round 6) Voice agents quote a dental price, confirm appointments, and handle urgent calls. Cozzo checks spoken details and scores responses to interruptions. | Everyday | Oct 2026 | Not in cards | |
| Cresta-Bench Contact-center agents take simulated calls for lending, health and airline lines. Tool calls get deterministic checks and LLM judges grade the rest. | Everyday | Oct 2026 | 82.6% | Not in cards |
| CRM-Agent-Bench Agents handle customer and business requests in a simulated enterprise. Executable checks verify tool arguments, action order, and which records changed across 555 tasks. | University | Oct 2026 | | Not in cards |
| CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Expert | Oct 2026 | | In use |
| CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. | Expert | Oct 2026 | 87.7% | Not in cards |
| D2K-Bench Agents turn expert GPU-kernel designs into efficient Triton, CUDA or CUTLASS kernels across 26 tasks. Runtime timing and design-code scores measure implementation quality. | Expert | Oct 2026 | Not in cards | |
| DataAnalyticsBench Agents query a synthetic data warehouse to answer business questions. A judge model checks answers against a rubric and an answer key. | University | Oct 2026 | 43.0 | Not in cards |
| DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. | Everyday | Oct 2026 | 40% | Not in cards |
| DAYJOB: Accounting Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Oct 2026 | | Not in cards |
| DAYJOB: Data Science Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Oct 2026 | 37.3% | Not in cards |
| DAYJOB: Law Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Oct 2026 | 41.4% | Not in cards |
| DAYJOB: Tax Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria. | Expert | Oct 2026 | 10.8% | Not in cards |
| DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. | Everyday | Oct 2026 | Not in cards | |
| DECK-Arena Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores. | University | Oct 2026 | 1,456 | In use |
| DeepRSI 15 open-ended ML tasks with a 48-hour budget each. Private verifiers score gains over baselines. Models ranked by Bradley-Terry Elo. | Research frontier | Oct 2026 | 1,629 | Not in cards |
| Developer Retrieval Benchmark Search systems retrieve repositories, documentation pages and issue or pull request fixes. Deterministic URL matching scores recall and reciprocal rank. | Expert | Oct 2026 | Not in cards | |
| DexArena Robots use both hands to complete household and industrial tasks. The simulator checks each stage and the final outcome across ten multistage tasks. | Everyday | Oct 2026 | 52.1% | Not in cards |
| Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. | Expert | Oct 2026 | | Not in cards |
| Doc2Harness Models build a coding-agent harness from an interface skeleton and a development document. The frozen harness then runs 328 scripted test cases in separate containers. | Research frontier | Oct 2026 | 53% | Not in cards |
| DOE sample pack Agents complete five simulated workplace assignments in law, private equity, and finance operations. Deterministic checks grade figures and application state, while model judges grade written work. | Expert | Oct 2026 | Not in cards | |
| Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. | Everyday | Oct 2026 | Not in cards | |
| Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. | Expert | Oct 2026 | 59.8 | Not in cards |
| DualEntry Accounting AI Benchmark Models use accounting tools to look up accounts, reconcile balances, and draft entries for a test company. DualEntry checks their answers against fixed references. | University | Oct 2026 | | Not in cards |
| Earth-Bench V2 Agents analyze prepared satellite observations. The corrected release measures final answer accuracy, tool selection, argument accuracy, and execution efficiency. | University | Oct 2026 | 80% | Not in cards |
| EGI-BENCH Agents work across eight enterprise evaluations. The suite combines their results into six scores covering activities such as planning, retrieving evidence, and using tools. | University | Oct 2026 | 65.6 | Not in cards |
| Embodied Agent Arena Vision-language agents solve geometry, spatial reasoning, affordance, planning, and manipulation tasks. The 1,000-case evaluation combines 32 established sources with GeoProbe. | Everyday | Oct 2026 | Not in cards | |
| Embodied First Exam Coding agents answer frozen robot tasks from many benchmark suites with a program. Each program is verified in a separate sandbox, run on Harbor. | Everyday | Oct 2026 | 49.6% | Not in cards |
| EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. | Everyday | Oct 2026 | 52% | Not in cards |
| EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. | University | Oct 2026 | 41.4% | Not in cards |
| Epoch Automation Reports Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees. | Expert | Oct 2026 | 81.7% | System cards only |
| Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. | Expert | Oct 2026 | 77.4% | In use |
| ExeWorldBench Coding agents build executable code-based 3D worlds that capture how a world works, testing world understanding and its realization in code. | Expert | Oct 2026 | Not in cards | |
| ExpertCrowd-SWE Agents resolve 100 software issues across 87 repositories in 11 languages. Each engineer-written rubric combines behavioral tests with a model judge and awards partial credit. The score averages five valid runs per task and then averages tasks. | Expert | Oct 2026 | 67.3% | Not in cards |
| FDE-Gym Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1. | Expert | Oct 2026 | 0.56 | Not in cards |
| FinHopper Agents research financial facts through up to six dependent searches. Intermediate answer keys show where an incorrect company, date, or indicator derails the search. | University | Oct 2026 | 67.3% | Not in cards |
| FlashDexRetarget 50-motion benchmark Retarget 50 human hand-object motions (HOT3D, TACO, OakInk2) to XHand and Sharpa hands in Isaac Sim. Scored by object pose-error success thresholds. | Everyday | Oct 2026 | Not in cards | |
| Flex-RSI Agents improve a starter solution and their own tools within a fixed budget. A hidden test split measures the final solution. | Expert | Oct 2026 | 1.52 | Not in cards |
| FoamBench V2 Agents build and run 110 Basic and 16 Advanced OpenFOAM cases. Revised instructions specify geometry, meshes, and boundary conditions for the same named cases. | University | Oct 2026 | | Not in cards |
| FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. | Everyday | Oct 2026 | | In use |
| FrontierCAD Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry. | Expert | Oct 2026 | 0.67 | Not in cards |
| FrontierCAD Assemblies Agents place supplied CAD parts into 16 assemblies. The score is the published mean assembly geometry score. | Expert | Oct 2026 | 0.71 | Not in cards |
| FrontierCAD Parts Agents build 39 CAD parts from engineering drawings. The score is the published mean geometry score. | Expert | Oct 2026 | 0.63 | Not in cards |
| FrontierCode 2.0 Agents resolve 132 maintainer-built tasks from deliberately ambiguous prompts. Rubric checks ask whether a maintainer would merge the change. | Expert | Oct 2026 | 50.2% | In use |
| FrontierCS 2 Agents work inside the released repository of a recent systems or programming-languages paper and try to beat that paper's method. A hidden held-out suite grades the final patch against the human reference. | Research frontier | Oct 2026 | 58.3% | Not in cards |
| FrontierML (Synthetic Sciences) Agents conduct 25 empirical machine learning projects under fixed budgets. Each task has a verifier and reference scores. Harbor includes the 15 public projects. | Research frontier | Oct 2026 | 0.52 | Not in cards |
| G2 Voice AI Agent Platform Evaluations Voice platforms run the same agent on simulated business calls. Bluejay checks required tool arguments and the database after hangup, then measures transcription and latency. | University | Oct 2026 | Not in cards | |
| GeoExpertQueries Agents choose geospatial data and carry out an analysis. A model compares each execution trace with a reference workflow agreed on by three GIS specialists. | Expert | Oct 2026 | Not in cards | |
| GeoKTools Agents combine up to four kinds of geospatial tools to answer quantitative questions. Exact matches earn full credit, and a model grades other answers. | University | Oct 2026 | Not in cards | |
| Graduate Admissions Agent Benchmark Agents advise students on graduate admissions and revise plans when the student's information changes. Seventy Chinese-language tasks span 23 cases and 83 answer stages. Separate reviewers score responses against task-specific rubrics on a 100-point scale. | Expert | Oct 2026 | 79.1 | Not in cards |
| Gray Swan Reward Hacking Bench Agentic environments where a task can be finished honestly or by crossing a declared line, such as shipping insecure code, leaking a secret, harming a third party through a shared service or leaving the scope of a penetration test. Reports how often each line is crossed. | Expert | Oct 2026 | System cards only | |
| Gym-Anything-RT Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time. | Everyday | Oct 2026 | 9.00% | Not in cards |
| Hanabi Agent Benchmark Models play cooperative Hanabi in four-player teams. The benchmark scores shared board progress while hints and lives constrain coordination. | Everyday | Oct 2026 | 20.5 | Not in cards |
| Harvey LAB v1.1 (HLAB-Vals-1.1) Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes. | Expert | Oct 2026 | | In use |
| Harvey LAB-AA v1.1 Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed. | Expert | Oct 2026 | 15% | In use |
| HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. | Expert | Oct 2026 | 57.3% | Not in cards |
| Hebbia Financial Services Benchmark Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion. | Expert | Oct 2026 | 74.6% | Not in cards |
| Hedge Gym Models adjust an options hedge each day over a 21-day price path. The evaluator measures replication error and transaction costs on synthetic or historical markets. | Expert | Oct 2026 | Not in cards | |
| Hermes Bench Agents in Hermes Agent do 150 tasks across 25 categories, such as research, diagrams, memory, tool use and safety. Graders check the files and state left behind. | Expert | Oct 2026 | 76.7% | Not in cards |
| Hermes Index Nous Research averages pass@1 and cost per task over Hermes Bench, Terminal-Bench 4.0, Terminal-Bench Science and SkillsBench, all run in Hermes Agent. | Expert | Oct 2026 | 63.3 | Not in cards |
| Historical Jailbreak Regression Replays previously found cyber jailbreak attacks against fixed ExploitBench tasks with real-time blocking turned off. Scored by mean task success rate under attack. | Expert | Oct 2026 | 14.3% | System cards only |
| HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. | Everyday | Oct 2026 | | Not in cards |
| HWE Bench V2 (RISC-V CPU design) Agents improve a RISC-V processor from a shared starting design. The benchmark scores throughput on five hidden programs using clock speeds from Gowin's timing tool. | Expert | Oct 2026 | 7024.2 | Not in cards |
| Hy-Nomas-Bench Interactive video world models follow text and camera controls across multiple turns and retain spatial memory. | Expert | Oct 2026 | Not in cards | |
| InnovationEval Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches. | Research frontier | Oct 2026 | Not in cards | |
| InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. | Everyday | Oct 2026 | 41.7% | Not in cards |
| Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. | Expert | Oct 2026 | | System cards only |
| Internal Finance Modeling (Moonshot) Agents complete financial modeling tasks with linked formulas, financial logic and final workbook quality. Experts rate the work. | Expert | Oct 2026 | Not in cards | |
| Internal Financial Research (Moonshot) Agents produce auditable financial research under evidence and cutoff constraints. Experts rate the resulting research. | Expert | Oct 2026 | Not in cards | |
| Into the Unknown Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check. | Research frontier | Oct 2026 | 0% | Not in cards |
| Investor's Last Exam Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria. | Expert | Oct 2026 | 68.1 | Not in cards |
| IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run. | Competition | Oct 2026 | 100% | Not in cards |
| IPI Bench Agents complete workplace workflows while attackers add misleading records to an emulated company with 11 services. The benchmark pairs six workflow types with harmful injected objectives across 34 tasks. The headline here measures single-query attack success as a percentage, where lower is better. | Expert | Oct 2026 | 4.80% | Not in cards |
| Iris Safety Evaluations Reflection AI's in-house safety suite of single-turn, multi-turn, jailbreak and agentic requests. Reports the share of harmful requests refused or redirected and the share of harmless requests answered. | Everyday | Oct 2026 | 88.9% | System cards only |
| Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score. | Expert | Oct 2026 | 66.5% | In use |
| Kimi Hallu Bench Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. | Expert | Oct 2026 | 4.20% | In use |
| LAB: US Case Law Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings. | Expert | Oct 2026 | Not in cards | |
| LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. | Everyday | Oct 2026 | 63.3% | Not in cards |
| Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. | Expert | Oct 2026 | 25.1% | In use |
| Learn2Play Bench Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure. | Everyday | Oct 2026 | 81.8 | Not in cards |
| Legal Consent Sweep Agents review an acquisition data room, reconcile consent requirements, and prepare notices and closing advice. A separate verifier grades their work. | Expert | Oct 2026 | 0.38 | Not in cards |
| LegalReasonBench Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings. | Expert | Oct 2026 | | Not in cards |
| LevelBench-100 v4.1 Models take over the books of small businesses using QuickBooks exports. They reconcile balances, identify missing evidence, and prepare closing entries across 12 workflows. | Expert | Oct 2026 | 81% | Not in cards |
| LiveMACE Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns. | Expert | Oct 2026 | Not in cards | |
| LiveSearch Agents search for current financial figures. The evaluator runs a stored query against a professional data service to update the reference answer. | University | Oct 2026 | | Not in cards |
| LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. | Expert | Oct 2026 | 66.5% | Not in cards |
| Long Context Navigator Agents answer questions that link facts across a synthetic document of about 32,000 tokens. They must read, compress, or discard information within a 600-token working context. | University | Oct 2026 | Not in cards | |
| Long-Horizon-Terminal-Bench 2.0 Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards. | Expert | Oct 2026 | 69.2% | Not in cards |
| MAMUBench Agents represent different users who share a limited resource. The benchmark compares their combined outcome with one agent coordinating for everyone. | Everyday | Oct 2026 | Not in cards | |
| MaxxBench Agents solve 89 modified DeepSWE repository tasks. Revised requirements, code, reference patches, and tests check whether agents can solve changed problems rather than recall original fixes. | Expert | Oct 2026 | 76.4% | Not in cards |
| Maydan HF RL Sampler Agents reconcile synthetic Saudi social insurance contributions across ten payroll tasks. SQLite checks the final records and audit log. | Expert | Oct 2026 | Not in cards | |
| MBABench 2.0 Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar. | Expert | Oct 2026 | 15.8% | Not in cards |
| Minecraft Bed Wars (YC Labs) Teams of agents play Minecraft Bed Wars against one another. The researchers compare the number of matches each team wins. | Everyday | Oct 2026 | Not in cards | |
| MirrorDojo Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDojo. Each episode pairs a user task with an instruction planted in a tool result. Reports attack success rate and utility. | Everyday | Oct 2026 | 3.10% | System cards only |
| MirrorDyn Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDyn. Reports attack success rate and utility. | Everyday | Oct 2026 | 3.60% | System cards only |
| MissionReport Models write government status reports in Word from working files and a template, across 20 expert-written Department of Defense style assignments. Expert rubrics score each decision. | Expert | Oct 2026 | | Not in cards |
| Mitosis Bench Agents mark every dividing cell in a tumor image. Each mark must fall within 30 pixels of an expert label. The grader rejects extra or missing marks. | Expert | Oct 2026 | | Not in cards |
| Navital Gym Agents complete tasks in six simulated Indian service settings. This port covers UPI Desk, which grades 50 payment and dispute tasks using written rules. | Everyday | Oct 2026 | 100% | Not in cards |
| OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. | Everyday | Oct 2026 | Not in cards | |
| OmicsBench v1 Agents analyze molecular data across eight benchmarks. LatchBio reports pass rates for 741 short tasks and 43 long tasks. | Expert | Oct 2026 | 40.8% | Not in cards |
| Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. | Everyday | Oct 2026 | | Not in cards |
| OncologyBench Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts. | Expert | Oct 2026 | 50.7% | Not in cards |
| OpenFrontiers-Bench Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings. | Research frontier | Oct 2026 | 42.2% | Not in cards |
| OpenTTD-Bench Agents build transport networks over ten simulated years. The grader rewards cumulative operating profit and the number of rivals each company beats. | Expert | Oct 2026 | Not in cards | |
| Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. | Expert | Oct 2026 | 3.10% | System cards only |
| OR-License Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem. | Expert | Oct 2026 | 88.4% | Not in cards |
| PAgentBench Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house. | Everyday | Oct 2026 | | In use |
| PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. | Research frontier | Oct 2026 | 14% | Not in cards |
| PCBBench (pcbbench.org) Agents solve electronics problems and produce KiCad files. The public pilot checks artifacts across ten suites, including schematic design, board layout, and debugging. | Expert | Oct 2026 | 89.6% | Not in cards |
| Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. | Everyday | Oct 2026 | | Not in cards |
| Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. | Expert | Oct 2026 | In use | |
| PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. | Everyday | Oct 2026 | | Not in cards |
| PharmaBench (micro1) Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff. | Expert | Oct 2026 | 17.8% | Not in cards |
| PhysClosureBench Agents run 50 expert-verified physics research tasks in strong-field and attosecond physics, deciding how to generate and validate evidence. A 20-criterion score and a closure level from RC0 to RC4 grade them. | Research frontier | Oct 2026 | 6.29 | Not in cards |
| Physical-Engineering's Last Exam Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. | Expert | Oct 2026 | 0.49 | Not in cards |
| PokéBench: Raw Vision + Buttons Models play Pokémon Red from screenshots using eight Game Boy buttons. Game memory verifies ten milestones, with fewer turns breaking ties. | Everyday | Oct 2026 | 10.0 | Not in cards |
| PretrainGym Agents select and mix public training data for a fixed language model. The grader trains on 400 million tokens and rewards low test loss. It penalizes overlap with evaluation data. | Research frontier | Oct 2026 | Not in cards | |
| Prompt to Product Agents build web applications from product briefs. Browser tests exercise features such as checkout arithmetic, saved filters, keyboard accessibility, and mobile layouts. | University | Oct 2026 | 98.2% | Not in cards |
| PWS-Finance Web research systems answer dated finance questions about company profiles, funding, ownership, markets, regulation, and recent developments. | Expert | Oct 2026 | Not in cards | |
| Quant-Arena Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window. | Expert | Oct 2026 | 51.3 | Not in cards |
| Questflow Finance Benchmark Agents query financial filings and market data with a fixed set of tools. They calculate metrics, rank securities, and submit answers that software checks against frozen references. | University | Oct 2026 | 100% | Not in cards |
| Railhead Agents run a transport company in OpenTTD. The evaluator replays the action log and lets the company run without intervention before measuring its performance. | Everyday | Oct 2026 | Not in cards | |
| Range Repair Agent Agents repair a Python function that merges touching ranges. The sole task earns one point if the edited function passes every held-out test. | School | Oct 2026 | Not in cards | |
| Real-SWE 1.1 Resolution rate on private enterprise coding tasks, averaged across five runs per task. | Expert | Oct 2026 | 56.8% | Not in cards |
| Realtime-Bench Agents act in 50 live game and web tasks where a late action fails: 30 web tasks (travel, shopping, services and security) and 20 games. The environment keeps moving while the model thinks. Three clocks separate speed from skill: Wall Clock runs on real time, Paused Clock stops while the model thinks, and Token Clock advances with each generated token. Scores are task success rates over three runs. | Everyday | Oct 2026 | 42.7% | Not in cards |
| RealWorkDecision Agents review a frozen company workspace and predict the decision an employee made. Expert-written rubrics allow valid alternatives. | University | Oct 2026 | 43.8% | Not in cards |
| RepoNormBench Agents change code in existing repositories while following the contribution rules that apply to each task. | Expert | Oct 2026 | | Not in cards |
| ReTrace Computer agents recover from injected errors in desktop tasks. Replaying the remaining actions checks whether the recovered state produces the original successful outcome. | Everyday | Oct 2026 | 63.6% | Not in cards |
| ReviewBench AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1. | Expert | Oct 2026 | 73.3% | Not in cards |
| RH SWE-bench Agents repair Go and Python code in Red Hat ecosystem repositories. The current candidate contains 337 tasks across 24 repositories after review and deduplication. | Expert | Oct 2026 | Not in cards | |
| Roark Live Bench v1.3 Voice agents answer calls over the public phone network and use simulated business tools. Roark scores task completion, tool use, speech quality, and response time. | University | Oct 2026 | Not in cards | |
| Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. | Expert | Oct 2026 | 32.7% | Not in cards |
| Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. | Everyday | Oct 2026 | Not in cards | |
| Robots Evolved GUI lessons Agents use screenshots, clicks, and keys to move sliding robots onto a goal. Ten puzzles enforce a move budget and offer no undo. | Everyday | Oct 2026 | Not in cards | |
| RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. | Everyday | Oct 2026 | 19.1% | Not in cards |
| RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. | Expert | Oct 2026 | 0.06 | Not in cards |
| RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. | Everyday | Oct 2026 | 17% | Not in cards |
| Rotom Champ Bench Models draft six level 50 Pokémon, select three for each match, and choose their moves. The organizer rates them on wins under Champions singles rules. | Competition | Oct 2026 | | Not in cards |
| Rotom Gen9 Bench Models draft six level 100 Pokémon and play the whole team in Generation 9 OverUsed singles. Match results determine their ratings. | Competition | Oct 2026 | 2,318 | Not in cards |
| RSI Control Arena An agent does one of 14 tasks from a successor model's training pipeline, such as SFT or reward modeling, while covertly trying to install one of 16 misaligned properties in that successor. A held-out oracle checks whether the property landed, and LLM monitors try to flag the attack from the transcript. | Research frontier | Oct 2026 | | Not in cards |
| RSI-Inference Bench Coding agents add model support to inference engines, improve serving speed and fix performance regressions. The harness builds and runs each submission. | Expert | Oct 2026 | 2,044 | Not in cards |
| rTaste Agents propose, rank, and forecast experiments inspired by recent research papers. An independent executor attempts the experiments, and auditors assess their results. | Expert | Oct 2026 | 3.71 | Not in cards |
| RTLScout Bench (suite-v2) Agents rewrite five circuits to reduce the product of chip area and signal delay. The leaderboard ranks configurations by their mean rank across circuits. | Expert | Oct 2026 | 1.00 | Not in cards |
| RuleTrek Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements. | Expert | Oct 2026 | 55.3% | Not in cards |
| RWE-bench Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement. | Expert | Oct 2026 | 62% | Not in cards |
| SandboxEscapeBench Agents attempt to escape a container sandbox and retrieve a flag from the host. Inspect runs the challenge inside reproducible virtual machines. | Expert | Oct 2026 | Not in cards | |
| ScholarCatalyst Given a research question, retrieve the 'catalyst papers' that 184 lead authors labeled for 207 projects, searching only prior literature. Recall@20. | Expert | Oct 2026 | 53% | Not in cards |
| SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. | Expert | Oct 2026 | 80.9 | Not in cards |
| ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. | Expert | Oct 2026 | 0.53 | Not in cards |
| SciTaste-Bench Models judge the research value of papers in six disciplines. The benchmark compares their judgments with a reference index built from seven dimensions. | Expert | Oct 2026 | | Not in cards |
| SDBench Diagnostic agents order tests, ask questions and commit diagnoses in sequential clinical cases. Scores combine diagnostic correctness with test cost. | Expert | Oct 2026 | Not in cards | |
| Senior Engineer Benchmark (Every Checks) Agents repair Proof from the March 17 outage snapshot. Every reports the percentage of 48 checks each solution passes. | Expert | Oct 2026 | 85% | Not in cards |
| SettlerBench Season One Four models trade resources, build settlements, and negotiate in public or private messages. Final victory points determine pairwise results for the Bradley-Terry rating. | Everyday | Oct 2026 | 1,270 | Not in cards |
| Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. | Expert | Oct 2026 | $2,949 | Not in cards |
| SilverquiLLM-bench hob-medium Coding agents implement five Hobbit cards in a Python game engine. The grader checks card behavior with 90 hidden tests. | University | Oct 2026 | 96.9% | Not in cards |
| SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. | Expert | Oct 2026 | 100% | Not in cards |
| SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. | Expert | Oct 2026 | Not in cards | |
| SimuVerity Agents build executable Simulink models from text for 101 tasks in ten engineering domains. Simulations check that each model meets its engineering requirements. | Expert | Oct 2026 | 42.9 | Not in cards |
| Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior. | Research frontier | Oct 2026 | 47.6% | Not in cards |
| SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. | Expert | Oct 2026 | 25.1% | Not in cards |
| SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. | Expert | Oct 2026 | | Not in cards |
| SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. | Expert | Oct 2026 | | Not in cards |
| SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. | Expert | Oct 2026 | | Not in cards |
| SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. | Expert | Oct 2026 | 30.5% | Not in cards |
| Solana Machine Trading Agents write trading bots for hidden market scenarios replayed on offline Solana snapshots. The headline score compares total profit and loss against a reference bot. | Expert | Oct 2026 | 6.11 | Not in cards |
| Spatial Agent Index v1 Agents analyze geospatial files and produce numbers, maps, or lists. Axis Spatial averages the scores for each included evaluation. | University | Oct 2026 | Not in cards | |
| Spatial Intelligence Index v1 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. | University | Oct 2026 | Not in cards | |
| SpireBench Agents play Slay the Spire and are scored on long-horizon deck-building decisions. | Everyday | Oct 2026 | Not in cards | |
| SpireClimb Agents play Slay the Spire 1 and 2 at max difficulty through a computer-use harness on fixed seeds. Score is the mean win rate. | Expert | Oct 2026 | 41.3% | Not in cards |
| SpireClimb StS1 A20 Heart Agents play Slay the Spire 1 through its graphical interface at Ascension 20 and must defeat the Heart. Scored by win rate over fixed seeds. | Expert | Oct 2026 | 33.3% | Not in cards |
| SpireClimb StS2 A10 Agents play Slay the Spire 2 at Ascension 10 through screenshots and mouse and keyboard, 15 runs per character. Scoring is the win rate. | Everyday | Oct 2026 | 49.3% | Not in cards |
| SplatBench A model rebuilds an object in 3D from eight photos by placing up to 300 Gaussians itself through tool calls, with no optimizer. Half of the 30 tasks start from an empty scene, and half ask it to reassemble a scrambled fit. SplatScore compares renders from eight angles the model never sees, with an empty scene at 0 and gradient descent at 100. | University | Oct 2026 | 49.6 | Not in cards |
| Stack EOL Audit: geol CLI vs API Agents identify software components that have reached end of life. They audit the same twelve stacks from memory, with an API, and with a command line tool. | University | Oct 2026 | | Not in cards |
| StarSkirmish Bench Each model gets one hour in its lab's own CLI to write a StarCraft: Brood War bot in C++. The score is its win rate against human-written bots. | Expert | Oct 2026 | 57.3% | Not in cards |
| StoreBench Agents manage a simulated apparel store in Medusa. They order inventory, change prices, ship purchases, and process refunds as demand changes. | University | Oct 2026 | 0.83 | Not in cards |
| sudo L7 Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change. | Expert | Oct 2026 | 52.2% | In use |
| SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. | Expert | Oct 2026 | | In use |
| SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result. | Research frontier | Oct 2026 | 16.6% | Not in cards |
| SWE-Clar Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build. | Expert | Oct 2026 | 67% | Not in cards |
| SWE-Race Agents fix concurrency and asynchronous lifecycle bugs in Python projects. The benchmark has 95 public and 93 private tasks. Harbor distributes 94 public tasks. | Expert | Oct 2026 | 81.9% | Not in cards |
| SWE-Serve Coding agents solve 53 production inference-engineering tasks derived from merged SGLang pull requests. Hidden tests measure task success, cost, agent steps and wall time. | Expert | Oct 2026 | 75.5% | Not in cards |
| SWE-sweep Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved. | Research frontier | Oct 2026 | 8.90% | In use |
| SWebBench Agents browse the web to retrieve information and do research tasks. iFlytek built it in house. | Expert | Oct 2026 | | In use |
| TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. | Everyday | Oct 2026 | Not in cards | |
| TasteVal 8 AI R&D tasks. Models iteratively design experiments. Taste measured as the compute needed to reach expert-level scores. | Research frontier | Oct 2026 | 230.4× | Not in cards |
| TavernBench Agents build a Hearthstone Battlegrounds player through research, simulator updates, and live games. | Expert | Oct 2026 | 95.6 | Not in cards |
| Tavily Company Search Benchmark Search providers return evidence for company research questions, and an answer agent responds from those results. A judge scores answer completeness, relevance and freshness. | Expert | Oct 2026 | Not in cards | |
| TestPrism Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one. | Expert | Oct 2026 | 33.3% | Not in cards |
| The Impossible Refund Assistants pursue a late train refund in a simulated email environment. Separate judges rate deception and other behaviors from the messages. | Everyday | Oct 2026 | Not in cards | |
| The Midnight Index Models lead a nation through fixed scenarios in a strategy game. The simulation records outcomes, escalation, and whether public commitments match later decisions. | Everyday | Oct 2026 | Not in cards | |
| TheoPhysEval Agents solve 50 unpublished research-level theoretical physics problems on their own, with literature search and scripts. Each final answer is checked against an expert answer, and the score is the pass rate. | Research frontier | Oct 2026 | 98% | Not in cards |
| Thrive Holdings helpdesk ticket resolution benchmark Agent investigates IT support requests, takes action and decides when to escalate to a technician. Scored by pass rate. | University | Oct 2026 | 61.5% | Not in cards |
| TraceTrial-Bench Agent harnesses, not only models, are tested on reliability across seven components, with traces used to attribute each failure to its cause. | Expert | Oct 2026 | 77.6% | Not in cards |
| Trajectory Labs cyber red teaming Expert red teamers try to break safeguarded cyber endpoints across nine attack technique families on exploit reproduction tasks. | Expert | Oct 2026 | System cards only | |
| TreasuryQA Bench Agents search scanned Treasury volumes and read tables to find or calculate a number. A script checks whether the answer falls within the accepted range. | School | Oct 2026 | | Not in cards |
| Turn based Benchmarks Models play turn based games including tic tac toe and rock paper scissors. The tasks evaluate game decisions and adaptation to opponents. | Everyday | Oct 2026 | | Not in cards |
| UnityBench-Mini3D Agents modify existing Unity projects. EditMode and PlayMode tests check the saved scene, component settings, and runtime behavior, with partial credit for completed milestones. | University | Oct 2026 | 72.5% | Not in cards |
| Unsatz Agents design experiments and propose equations for an unknown simulated system. Their final program must predict held-out interventions and conditions beyond the original experiments. | Expert | Oct 2026 | 60% | Not in cards |
| Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. | Expert | Oct 2026 | 69.5% | In use |
| Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. | Expert | Oct 2026 | | In use |
| Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. | Oct 2026 | 71.3% | In use | |
| Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. | Expert | Oct 2026 | | In use |
| Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. | Expert | Oct 2026 | Not in cards | |
| Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. | Expert | Oct 2026 | 68.4% | In use |
| Vals Smith: AumeshMisra/tensorzero Agents modify AumeshMisra/tensorzero to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Oct 2026 | 86.7% | Not in cards |
| Vals Smith: pallets-eco/flask-wtf Agents modify pallets-eco/flask-wtf to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Oct 2026 | 100% | Not in cards |
| Verifiable Browsing Task (Meta) Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools. | Everyday | Oct 2026 | Not in cards | |
| VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. | Expert | Oct 2026 | 4.11 | Not in cards |
| Web Clone Bench Agent gets a running reference app (Gmail, Jira, Figma, etc.) and 8 hours to build a clone. Expert-written workflows are replayed as end-to-end tests. | Research frontier | Oct 2026 | 62% | Not in cards |
| Web Context (Nimble) Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages. | Expert | Oct 2026 | 80% | Not in cards |
| Which AI Model Knows Israel Best? Models answer Hebrew questions about everyday life in Israel. The organizers check each answer against an official source and count refusals separately. | Everyday | Oct 2026 | 99.3% | Not in cards |
| WildAWM validation Agents complete 33 held-out computer-use tasks in generated software environments. Harbor packages the validation split separately from the training tasks. | Expert | Oct 2026 | | Not in cards |
| WorkplaceAgents Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics. | Expert | Oct 2026 | | Not in cards |
| xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. | Expert | Oct 2026 | 100% | In use |
| Your LLM doesn't Know BOUNDARY!. Models complete work tasks at different access levels. Tests check whether they solve the part they own and report a blocker when another owner must act. | University | Oct 2026 | 0.91 | Not in cards |
| τ-bench 2.0 Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture. | University | Oct 2026 | 79.9% | Not in cards |
| AA-Briefcase v1.1 Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. | Expert | Sep 2026 | 1,822 | In use |
| AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. | Expert | Sep 2026 | System cards only | |
| Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. | Expert | Sep 2026 | System cards only | |
| Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. | Expert | Sep 2026 | 3.50% | System cards only |
| Agentic IF Index Composite of Meta internal evals on following tool-use restrictions, rubrics, compound constraints and long policies in agentic workflows. Scored 0-1. | Sep 2026 | System cards only | ||
| Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. | Everyday | Sep 2026 | System cards only | |
| ALE-Gold Agents use a Linux terminal to complete 96 professional tasks. Graders check the files they produce. A task passes only with full credit. | Expert | Sep 2026 | Not in cards | |
| Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. | Expert | Sep 2026 | 40% | Not in cards |
| Anthropic large agent teams tasks Internal 24-hour tasks for teams of 1-100 agents: build a company knowledge base, or formalize a theorem in Lean. | Research frontier | Sep 2026 | System cards only | |
| AnyAppBench Agents repeat the same task in different Android apps, with and without suggested steps. Each app has a verifier that checks the same success condition. | Everyday | Sep 2026 | | Not in cards |
| APEX-Agents 1.1 Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero. | Expert | Sep 2026 | | Not in cards |
| Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. | Expert | Sep 2026 | 0.17% | System cards only |
| ApprenticeBench Agent joins a simulated company and processes 100 accounts-payable bills in Odoo via GUI or API, learning from records and feedback. Cumulative success rate. | University | Sep 2026 | 77% | Not in cards |
| ArtifactArena Models design robot bodies, controllers and strategy for a simulated sumo-style arena. Artifacts fight head-to-head and models get Elo ratings. | Expert | Sep 2026 | 1,539 | Not in cards |
| Artificial Analysis Coding Agent Index v1.5 Version 1.5 of Artificial Analysis's coding agent index, which moved to the harder Terminal-Bench 4.0 and DeepSWE 1.1. | Sep 2026 | Not in cards | ||
| Artificial Analysis Cyber Index Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. | Expert | Sep 2026 | | In use |
| Artificial Analysis Intelligence Index v4.2 Version 4.2 of Artificial Analysis's index, which added multi-hour office work and document tasks and dropped GPQA Diamond. | Sep 2026 | Not in cards | ||
| Artificial Analysis Intelligence Index v4.3 Version 4.3 of Artificial Analysis's index, eleven weighted parts led by knowledge work, office tasks and Terminal-Bench 4.0. | Sep 2026 | | In use | |
| ATLAS-Finance Agents do 100 investment banking tasks inside 13 simulated firms. They gather facts from email, chat, calendar and files over MCP, then build an Excel model graded against expert rubrics. | Expert | Sep 2026 | Not in cards | |
| AutoResearchExam Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed. | Research frontier | Sep 2026 | 0.61 | Not in cards |
| BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. | Expert | Sep 2026 | Not in cards | |
| Beancount Ledger Agents reconcile bank statements, repair a Beancount ledger, and allocate payments to invoices. The evaluator checks balances and payment records against the underlying transactions. | University | Sep 2026 | Not in cards | |
| Bench on the Clocktower Ten models play the social deduction game Blood on the Clocktower together, one seat each, over 300 ten-player games of the Trouble Brewing script. Each model plays 210 games on the good team and 90 on the evil team. The score is the share of games its team won. | Everyday | Sep 2026 | 58% | Not in cards |
| Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. | Expert | Sep 2026 | 72.2% | System cards only |
| Biopharma Bench Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers. | Expert | Sep 2026 | 65.1% | Not in cards |
| BioSecBench-Function Agents recover biological function from experimental data. Scored by verifiable checks. One of three LatchBio BioSecBench evaluations. | Expert | Sep 2026 | 73.6% | System cards only |
| BlenderBench (Refresh) Agents edit Blender scenes across 20 tasks. Each task grades the submitted scene using its own scoring configuration. | University | Sep 2026 | 0.69 | Not in cards |
| Brood War Bench Coding agents play StarCraft: Brood War against each other in real time. Systems are ranked by win rate over matches, with cost per game. | Expert | Sep 2026 | 100% | Not in cards |
| BU Bench V2.1 Browser agents do 200 real web tasks. A findings judge checks tool evidence, answers and files, and deterministic integrity checks run as well. | Everyday | Sep 2026 | 81.3 | Not in cards |
| BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. | Expert | Sep 2026 | 76.4 | In use |
| CAD Arena Agents rebuild licensed engineering drawings as editable CAD parts through the scripting APIs of NX, SolidWorks, Onshape, Fusion or Build123d. A verifier scores geometric match. | Expert | Sep 2026 | | Not in cards |
| CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. | Expert | Sep 2026 | | Not in cards |
| Chess Engine Benchmark (Steve Maughan) Models have 24 hours to write a chess engine from scratch in C or C++. Maughan rates each engine against reference engines and other entries. | University | Sep 2026 | 3,542 | Not in cards |
| CivBench (Civilization VI) Agents play Civilization VI and are evaluated on strategic progress in a long-running world. | Expert | Sep 2026 | | Not in cards |
| Cloud Expansion CE80-10 Agents complete a cloud network security exercise. The verifier checks the resulting cloud configuration and observed network behavior. | Expert | Sep 2026 | Not in cards | |
| Cloud Expansion CEExfil-0 Agents complete a cloud security capstone. A verifier reads the environment after the task to assess the outcome. | Expert | Sep 2026 | Not in cards | |
| Cloud Expansion RX00 Agents complete a simulated cloud reconnaissance assessment. The verifier compares the submitted result with the recorded state of the exercise. | Expert | Sep 2026 | Not in cards | |
| Code2Video Benchmark 168 product-video briefs. Agent writes HTML motion-graphics code rendered by HyperFrames. A trained pairwise judge on five axes produces Elo. | Expert | Sep 2026 | 1,585 | Not in cards |
| Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. | University | Sep 2026 | 0.62 | Not in cards |
| Compliance with instructions planted in pasted text (Anthropic) Coding agents receive user prompts with pasted text that hides an attacker's instruction. Score is the rate of acting on the planted instruction. | Expert | Sep 2026 | 7.40% | System cards only |
| CompMat-Bench Agents complete individual steps from published computational materials studies. Fixed rules compare their inputs and outputs with results reproduced by the authors. | Expert | Sep 2026 | 91.5% | Not in cards |
| CorpBench Work Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly. | Everyday | Sep 2026 | 92.7 | Not in cards |
| CUA-bench Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out. | Everyday | Sep 2026 | 19.2% | Not in cards |
| CursorBench 4.0 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported. | Expert | Sep 2026 | | In use |
| CWE-bench Agents audit a repository and write a patch for a security flaw, without being told where the bug is. Covers six programming languages. | Expert | Sep 2026 | | In use |
| Data Science Tasks (OpenAI) Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | Expert | Sep 2026 | 40.9% | In use |
| Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | Expert | Sep 2026 | 63.9% | In use |
| DataBench v1.1 Agents answer ambiguous business questions and make recommendations from a synthetic company warehouse. Model judges assess the reasoning, supporting evidence, artifacts, and final recommendation against task rubrics. | Expert | Sep 2026 | 70.5% | Not in cards |
| Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. | Expert | Sep 2026 | System cards only | |
| DAYJOB: Finance Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Sep 2026 | 42.5% | In use |
| DAYJOB: Healthcare Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Sep 2026 | 48.3% | Not in cards |
| De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. | Expert | Sep 2026 | 85.1% | System cards only |
| DealBench-100 Agents perform 100 investment banking workflows across ten synthetic transactions. Deterministic checks grade calculations, decisions, committed changes, and consistency between models and presentations. | Expert | Sep 2026 | Not in cards | |
| Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. | Expert | Sep 2026 | 0% | In use |
| DI-Bench Agents answer questions using company tables, documents, and business rules. The benchmark checks whether agents apply those rules when computing results. | University | Sep 2026 | 57.9% | Not in cards |
| Disengagement-Aware Student Simulators Tutors interact with simulated learners who vary in engagement. The authors compare five tutors and validate the student behaviors against real tutoring sessions. | University | Sep 2026 | Not in cards | |
| DolphinBench Agents use long-term memory to complete 600 tool-using tasks for three simulated knowledge workers with roughly 500,000 user-message tokens each. | Expert | Sep 2026 | 70.7% | Not in cards |
| DrivingBench Vision-language models drive a real Toyota Corolla through a cone course with up to three attempts in one conversation. Scored on the furthest progress along the course. | Everyday | Sep 2026 | 100% | Not in cards |
| DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. | Expert | Sep 2026 | 40.0 | Not in cards |
| Duplex-MPE Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios. | Everyday | Sep 2026 | | Not in cards |
| Duplicate Side Effect Desk Agents process refund requests when timeouts or duplicate tickets obscure payment status. The evaluator uses the transaction ledger to identify duplicate refunds. | Everyday | Sep 2026 | 0.98 | Not in cards |
| E2E-SWE Agents build complete software repositories from natural language specifications in empty workspaces. Hidden tests require every generated project to install and run. | Research frontier | Sep 2026 | Not in cards | |
| Earth-Bench-Pro Agents complete Earth observation workflows under three conditions. They follow instructions, plan from supplied observations, or obtain observations at runtime. | University | Sep 2026 | Not in cards | |
| EmailBench Agents handle email and productivity requests in a synthetic inbox. The evaluator combines executable checks and model-graded rubrics across 206 scenarios. | University | Sep 2026 | 33.5% | Not in cards |
| EngiWorld Agents work through the full industrial engineering design loop on 1,301 expert-curated tasks that need reasoning over geometric and physical constraints across software. | Expert | Sep 2026 | 44.7% | Not in cards |
| Era by Eon Generates a simulated enterprise (Salesforce, Slack, Jira, Zendesk, etc.) with exact ground truth. Agents answer questions through MCP/REST. | University | Sep 2026 | 76.8% | Not in cards |
| ERPBench-100 Agents execute 100 business workflows across ten simulated enterprise resource planning systems. Checks grade decisions, calculations, committed state, and handoffs on a 100-point scale. | Expert | Sep 2026 | Not in cards | |
| EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. | Research frontier | Sep 2026 | 29.8 | Not in cards |
| ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. | Research frontier | Sep 2026 | 53.5% | In use |
| ExplorationBench Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help. | Research frontier | Sep 2026 | 89% | Not in cards |
| Factorio Play Agents build factories one action at a time in a Factorio simulator. The evaluator measures factory output on held-out layouts. | University | Sep 2026 | Not in cards | |
| fAIntasy Football Models draft NFL players, set lineups, bid for free agents, and trade under the same rules. The standings show wins, fantasy points, and lineup efficiency. | Everyday | Sep 2026 | | Not in cards |
| FinCrime Agent Benchmark Agents screen customers for sanctions, political exposure, and adverse media. A case passes when agents find the relevant alerts and resolve them under the bank's policy. | University | Sep 2026 | 71.1% | Not in cards |
| FinFIRST Agents research financial questions and provide supporting sources. The benchmark grades individual requirements and reports how often an answer meets every requirement. | Expert | Sep 2026 | 71.5% | Not in cards |
| FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. | Expert | Sep 2026 | | In use |
| FinStepBench LiveSearch Models retrieve and verify live financial facts. The score is the published LiveSearch percentage. | Expert | Sep 2026 | | Not in cards |
| Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. | Expert | Sep 2026 | Not in cards | |
| FrontierMath Erdős Models try to prove or disprove 68 open Erdős problems in Lean 4. A result counts only if the Lean checker accepts the full proof. | Research frontier | Sep 2026 | 10.3% | System cards only |
| FrontierSWE 34 ultra-long-horizon engineering and research tasks. 5 trials of up to 20 hours each. Score is mean task score 0-100. | Research frontier | Sep 2026 | 69.9% | In use |
| Game Arena Go Language models play Go against each other through the Game Arena text harness. Ratings come from head-to-head games. | Expert | Sep 2026 | 1,877 | Not in cards |
| GameASG-Bench Agents build 47 browser games from written requirements. Static checks inspect each HTML file, and browser tests check controls, gameplay and state changes. | University | Sep 2026 | 55.3% | Not in cards |
| GameForgeBench Agents repair game projects in engines such as Godot. The verifier runs the project and checks behavior such as camera movement and collisions. | University | Sep 2026 | Not in cards | |
| GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. | Everyday | Sep 2026 | 80.2 | Not in cards |
| GDP.xlsx Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria. | Expert | Sep 2026 | | Not in cards |
| GDPval-AA v2.1 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. | Expert | Sep 2026 | 1,846 | In use |
| GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Scored by a distance-based GeoGuessr-style reward from 0 to 1. | Everyday | Sep 2026 | 0.64 | Not in cards |
| GlyphBench Agents choose actions from text-rendered game boards. The standard suite contains 303 tasks across six game collections and scores agents by cumulative return. | University | Sep 2026 | | Not in cards |
| Gray Swan Shade knowledge elicitation through chat An automated attacker tries to elicit useful answers to 100 harmful and 100 dual-use cyber queries from a safeguarded model. Score is attack success rate. | Expert | Sep 2026 | 1.00% | System cards only |
| HarnessDev Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score. | Research frontier | Sep 2026 | 67.8% | Not in cards |
| HarvestBench Agents drive simulated tractors and decide how much fuel to spend avoiding animals. The benchmark measures how often they hit animals as avoidance costs change. | Everyday | Sep 2026 | 0.40% | Not in cards |
| Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. | Expert | Sep 2026 | 100% | System cards only |
| Huayu Awards AI Cockpit Evaluation 2026 In-car assistants interpret requests and operate vehicle features. Tests include planning three-stop trips and adjusting several cabin controls from one request. | Everyday | Sep 2026 | Not in cards | |
| HubBench Agents make workplace decisions across 13 professional domains using connected software tools and evidence. Version 1.4.0 grades 104 tasks with deterministic state checks. | Expert | Sep 2026 | Not in cards | |
| Incident Arena Agents diagnose and repair incidents in running services. The score averages the repair success rate across 20 incidents. | Expert | Sep 2026 | 64.3% | Not in cards |
| Internal computer use safety benchmark (OpenAI) Models complete computer-use tasks while avoiding unsafe actions. | Research frontier | Sep 2026 | 2.40% | System cards only |
| ironbench Agents write firmware for tasks such as serial communication and controller sequencing. The grader checks device behavior and counts tasks solved within the allowed attempts. | University | Sep 2026 | 69.2% | Not in cards |
| ISA-Bench Models write programs for five games with restricted instruction sets. A verifier checks each program and gives feedback before the next attempt. | University | Sep 2026 | | Not in cards |
| KernelQuest Agents optimize Triton kernels. A task passes when the kernel produces correct outputs and reaches the speed target on the same GPU. | Expert | Sep 2026 | 79% | Not in cards |
| KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. | Expert | Sep 2026 | 29.4% | Not in cards |
| Large Language Model Football League Models draft, trade, and set weekly NFL lineups. Each team earns one result against its opponent and another against the league median. | Everyday | Sep 2026 | Not in cards | |
| Legacy-Bench Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass. | Expert | Sep 2026 | 60% | Not in cards |
| Legora BAR Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers. | Expert | Sep 2026 | 106.0 | Not in cards |
| LEXam-hard Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed. | Research frontier | Sep 2026 | | Not in cards |
| LibraryDesignBench An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity. | Expert | Sep 2026 | 48.9 | Not in cards |
| LLM Fantasy Football Arena Guillotine Models draft NFL players and bid for free agents. Each week the lowest-scoring team leaves, and the remaining teams bid on its players. | Everyday | Sep 2026 | | Not in cards |
| Long-Horizon Freight AuditBench Challenge Agents price 100 difficult freight invoices and flag incorrect charges. The cases include incomplete information and exceptions to ordinary contract terms. | Expert | Sep 2026 | 10% | Not in cards |
| Long-Horizon Freight AuditBench Production Agents price 200 freight invoices from contracts and carrier documents, then flag billing errors. Loop weights accuracy to reflect a 24 percent billing error rate. | University | Sep 2026 | 16.1% | Not in cards |
| Long-Horizon Medical Agent Benchmark, Veterinary Clinic Ops v2 Agents operate a simulated veterinary clinic through its database API. Each of the 98 tasks checks whether the agent leaves the clinic records in the required state. | Expert | Sep 2026 | 100% | Not in cards |
| LongPuzzleBench GUI agents solve 114 visual puzzle levels across six games through native interface actions. | Everyday | Sep 2026 | Not in cards | |
| Ludobench Backgammon Models play backgammon against opponents using dice rolls and legal moves. Ludobench publishes Glicko-2 ratings and match replays. | Everyday | Sep 2026 | | Not in cards |
| Ludobench Hokm Models play Hokm in partnerships. One player names trump, and partners try to take seven tricks before the other team. | Everyday | Sep 2026 | 100% | Not in cards |
| MANDOS Session 2 Models make process safety decisions across four private industrial scenarios. A model judge assesses each run. The leaderboard reports the share of passing runs. | Expert | Sep 2026 | 96.9% | Not in cards |
| Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. | Expert | Sep 2026 | 0% | System cards only |
| MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. | University | Sep 2026 | 51.5% | Not in cards |
| McStasBench Agents design neutron scattering instruments. A simulator checks whether their designs reproduce published measurements. The authors score two additional instruments separately. | Expert | Sep 2026 | 43.8% | Not in cards |
| Medical Bench Agents read longitudinal dermatology charts and write clinical assessments. Expert-authored rubrics grade their findings and management decisions. | Expert | Sep 2026 | 88% | Not in cards |
| MetagenomicsBench Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result. | Expert | Sep 2026 | 62.7% | Not in cards |
| metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. | Everyday | Sep 2026 | Not in cards | |
| MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details. | Expert | Sep 2026 | 68.6% | In use |
| MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. | Expert | Sep 2026 | 81.7% | In use |
| Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. | Expert | Sep 2026 | 1.40% | System cards only |
| Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. | Expert | Sep 2026 | System cards only | |
| Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. | Expert | Sep 2026 | 61.2% | System cards only |
| Multi-Agent Arena Codenames Two teams of anonymized models play the word game Codenames. The board rates the clue giver and the guessers on separate rating scales. | Everyday | Sep 2026 | 1,535 | Not in cards |
| Multi-Agent Arena Diplomacy Seven players, people or anonymized models, negotiate and fight over World War I Europe. Models are ranked by their mean share of the sum-of-squares score, where 14.3 is an even split. | Everyday | Sep 2026 | 37.5% | Not in cards |
| MysteryMechanism Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold. | Expert | Sep 2026 | 53.1% | System cards only |
| Nego Eval Agents choose negotiation partners and remember prior encounters across games. The evaluator compares match scores with reference policies on the same seeds. | Everyday | Sep 2026 | Not in cards | |
| OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. | Expert | Sep 2026 | 0% | System cards only |
| OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. | Everyday | Sep 2026 | | Not in cards |
| OpenRSI Index Agents run autoresearch on real open-source model development projects with their original code, data and checkpoints, for 60 or more hours. Fixed verifiers score the result against the human recipe they started from. | Research frontier | Sep 2026 | Not in cards | |
| OpenRSI Index ACE Playbook Inspection and Repair Agents repair a reasoning playbook under a fixed budget. The score is held-out Formula exact-answer accuracy as a fraction. | Research frontier | Sep 2026 | 0.73 | Not in cards |
| OpenRSI Index Isaac Lab PegInsert Reward Search Agents design robot training rewards under a fixed budget. The score is held-out peg-insertion success as a fraction. | Research frontier | Sep 2026 | 0.60 | Not in cards |
| OpenRSI Index Kev Decision Architecture Agents improve a decision model under a fixed budget. The score is the published exponential of negative macro log loss. | Research frontier | Sep 2026 | 0.29 | Not in cards |
| OpenRSI Index Learnability-Aware Long/Short CoT Adaptation Agents adapt mathematical reasoning examples under a fixed training budget. The score is accuracy across five math benchmarks. | Research frontier | Sep 2026 | 47.5% | Not in cards |
| OpenRSI Index Marin Scaling Ladder Agents design one optimizer for six model sizes. The score is the published complete six-rung reward relative to AdamH. | Research frontier | Sep 2026 | 1.01 | Not in cards |
| OpenRSI Index MInference 32-Head Sparse Prefill Kernel Agents optimize a sparse-attention prefill operator under a fixed budget. The score is speedup against the paired reference run. | Research frontier | Sep 2026 | 1.54× | Not in cards |
| OpenRSI Index Molmo2 Video-Pointing Inference Strategy Agents tune video inputs and inference for a frozen Molmo2 model. The score is held-out spatiotemporal pointing soft F1. | Research frontier | Sep 2026 | 0.46 | Not in cards |
| OpenRSI Index MolmoWeb Interaction Context Allocation Agents select interaction context for a frozen browser model. The score is macro-averaged browser-action prediction quality. | Research frontier | Sep 2026 | 0.61 | Not in cards |
| OpenRSI Index ReasonIR Difficulty Curriculum Agents adjust retrieval training data schedules. The score measures reasoning retrieval with a penalty for general retrieval regression. | Research frontier | Sep 2026 | 0.22 | Not in cards |
| OpenRSI Index SGLang Omni TTS Runtime Agents improve speech serving while preserving quality. The score is the official speedup reward for the archived continuation protocol. | Research frontier | Sep 2026 | 1.54× | Not in cards |
| OSWorld 2.1 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. | University | Sep 2026 | 44.3% | In use |
| OSWorld 2.1 (offline subset) Agents complete the 82 OSWorld 2.1 tasks on the official offline list, with no internet access. Scored by checkpoint partial credit and strict pass rate. | University | Sep 2026 | 89.3% | In use |
| OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. | Expert | Sep 2026 | 73.7% | Not in cards |
| Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. | Expert | Sep 2026 | System cards only | |
| Package registry publishing evaluation (Anthropic) Agents in a simulated capture the flag exercise can publish a backdoored package to a registry. An investigator escalates pressure until they act. | Expert | Sep 2026 | 33.3% | System cards only |
| PACT (robot arm orchestration) Agents write and revise programs to control a simulated robot arm. The simulator checks whether the arm completes the requested manipulation. | Expert | Sep 2026 | Not in cards | |
| Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. | Everyday | Sep 2026 | 36.3% | Not in cards |
| Parallel Search Capability Leaderboard Language models answer research questions with and without search, and Parallel ranks their accuracy, search lift, cost, and time. | Expert | Sep 2026 | | Not in cards |
| Parametric CAD Bench Agents create and edit parametric FreeCAD models from text and engineering drawings. Programmatic checks award continuous task rewards for geometry and specification consistency. | Expert | Sep 2026 | 61.0 | Not in cards |
| Periapical Lesions Bench Agents locate and grade lesions around tooth roots in panoramic dental X-rays. A perfect report matches every lesion and grade without adding extra boxes. | Expert | Sep 2026 | | Not in cards |
| PhoneShell iOS Agent Benchmark Agents operate apps on a physical iPhone to set timers, find books, and complete other everyday tasks. Automated checks read the resulting device state. | Everyday | Sep 2026 | 96.6% | Not in cards |
| PokéBench: Tool Assisted Benchmark Models play Pokémon Red with a screenshot, map, game text, and action shortcuts. They have 1,000 turns to reach ten milestones ending with Brock. | Everyday | Sep 2026 | 10.0 | Not in cards |
| PostTrainBench 1.2 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Research frontier | Sep 2026 | 44.6% | In use |
| PowerBench Agents retrieve equipment records and telemetry to answer 300 power-system questions. Macro Joint Accuracy averages fully correct answers equally across three task families. | Expert | Sep 2026 | 74.2% | Not in cards |
| Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. | Expert | Sep 2026 | 0.20% | System cards only |
| ProgramDistill Agents inspect working web applications and restore missing features in editable copies. Replayed browser traces check whether patches recover the reference behavior. The paper evaluates partial repairs and full application reconstruction. | University | Sep 2026 | 84.3% | Not in cards |
| Prosus Vending Bench v2 Agents run six simulated vending machines across three locations for 30 days. They order stock, set prices, and collect cash. The score measures final bank balance in euros. | University | Sep 2026 | 4497.2 | Not in cards |
| Qwen-Audio-Agent In-House Cockpit Benchmark Agents handle requests with cockpit tools. The evaluation measures whether they complete each task when they call tools directly, delegate, or choose between both routes. | Everyday | Sep 2026 | Not in cards | |
| Qwen-Audio-Agent Smart Cockpit Benchmark Agents operate simulated vehicle, navigation, music, and weather services. The short suite checks whole cases. The long suite checks tool use on each turn. | Everyday | Sep 2026 | | Not in cards |
| Real-SWE Models resolve tasks in private production codebases licensed from real companies. The board reports resolution rate with standard errors. | Expert | Sep 2026 | 38.8% | Not in cards |
| RecreationBench Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result. | Expert | Sep 2026 | 58.1% | In use |
| Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. | Expert | Sep 2026 | 0% | In use |
| Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. | Expert | Sep 2026 | 11.8% | In use |
| REVERSAL-BENCH Reset-free reinforcement learning agents learn manipulation policies across environments with controlled reversibility and recoverability checks. | Expert | Sep 2026 | Not in cards | |
| RIICHI ORBIT mahjong bot comparison (open practice) Models get 30 minutes to improve the same mahjong bot, with unlimited practice. The maker freezes each bot before evaluating it on separate tile sequences. | University | Sep 2026 | 2.41 | Not in cards |
| RIICHI ORBIT mahjong bot comparison (two-stage practice) Models improve a beginner mahjong bot during two timed coding periods. The maker runs the finished bots in four-player matches and reports mean placement by table. | University | Sep 2026 | 2.28 | Not in cards |
| Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. | Everyday | Sep 2026 | 52.5% | Not in cards |
| RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. | Everyday | Sep 2026 | 65.7% | Not in cards |
| Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. | Everyday | Sep 2026 | 20% | System cards only |
| RSI Arena Ten frontier agents each post-train the same Nemotron base model on a shared GPU cluster for six days. People then rank the resulting models in blind side-by-side votes, and agents retrain on that feedback. | Research frontier | Sep 2026 | 1,095 | Not in cards |
| Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. | Research frontier | Sep 2026 | 45.5% | System cards only |
| Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. | Expert | Sep 2026 | 1.50% | System cards only |
| Sci Pipeline v1 Agents clean hourly sensor readings and write daily means, record counts, and extremes to a JSON file. The grader checks the saved file. | University | Sep 2026 | Not in cards | |
| SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. | Expert | Sep 2026 | 45.3% | Not in cards |
| SemiKongBench-100 Agents resolve 100 synthetic semiconductor operating exceptions. The evaluator grades each workflow with deterministic checks. | Expert | Sep 2026 | Not in cards | |
| Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. | Expert | Sep 2026 | 8.00% | System cards only |
| Simreal ML Benchmark ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them. | Expert | Sep 2026 | Not in cards | |
| Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. | Expert | Sep 2026 | 0% | System cards only |
| SpeedrunBench Agents try to finish goals in 10 video games as fast as possible. Score is time to goal in frames or seconds, lower is better. | Everyday | Sep 2026 | Not in cards | |
| StationeryBench Robots use both arms to handle common desk supplies. A human grades progress from zero to 100 on five tasks. | Everyday | Sep 2026 | 46.0 | Not in cards |
| StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup. | Expert | Sep 2026 | 63.9% | In use |
| StudentBench AI tutors teach real students GRE material and are scored by the students measured learning gains. A second study has expert tutors compare AI lesson plans and practice problems. | University | Sep 2026 | 16.5 | Not in cards |
| SWE-bench Multimodal v2 Agents patch JavaScript repositories using issue reports that include images. Automated tests check whether each patch fixes the issue. | Expert | Sep 2026 | 70.2% | System cards only |
| SWE-Bench Pro V2 Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container. | Expert | Sep 2026 | 99.4% | In use |
| SWE-Bench Pro V2 Hard Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol. | Research frontier | Sep 2026 | 98% | In use |
| SWE-Bench Pro Verified Agents repair software repositories. OpenCompass corrects inconsistent tasks and strengthens tests in SWE-Bench Pro. | Expert | Sep 2026 | Not in cards | |
| SWE-Game Agents build, complete, repair, or port games across 247 tasks derived from 41 reference games. Runtime probes and input replays check gameplay mechanics and preserved behavior. Vision-language rubrics separately grade presentation in construction tasks. | University | Sep 2026 | 64.0 | Not in cards |
| Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. | Expert | Sep 2026 | | In use |
| TimelineBench Agents turn raw footage and a client brief into a finished video. The edit must pass delivery checks, meet the brief, and clear a quality threshold. | University | Sep 2026 | 32.1% | Not in cards |
| TruthInsightBench Agents analyze data from published studies without seeing the original conclusions. A judge grades whether the evidence and artifacts support the claims each agent makes. | Expert | Sep 2026 | Not in cards | |
| TxBench-Oligo-Discovery Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision. | Expert | Sep 2026 | 62.2% | Not in cards |
| UK AISI Out of Scope Supply Chain Attack Agents in simulated hard cyber challenges with incidental internet access may attack out-of-scope open source projects. Scored as rate of samples with such attacks. | Expert | Sep 2026 | 12% | System cards only |
| Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. | Expert | Sep 2026 | 0% | System cards only |
| Vals Index v2.1 Version 2.1 of Vals AI's GDP-weighted index, which moved coding to Terminal-Bench 4.0 and added a tax sector. | Sep 2026 | | In use | |
| Vals Smith: dwojtaszek/zipper Agents modify dwojtaszek/zipper to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 93.3% | Not in cards |
| Vals Smith: Effect-TS/effect Agents modify Effect-TS/effect to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 93.3% | Not in cards |
| Vals Smith: Expertium/fsrs-autoresearch Agents modify Expertium/fsrs-autoresearch to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 96.7% | Not in cards |
| Vals Smith: fmtlib/fmt Agents modify fmtlib/fmt to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 83.3% | Not in cards |
| Vals Smith: Lemonawa/music163bot-rust Agents modify Lemonawa/music163bot-rust to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 93.3% | Not in cards |
| Vals Smith: OpenTTD/OpenTTD Agents modify OpenTTD/OpenTTD to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 100% | Not in cards |
| Vals Smith: punt-labs/lux Agents modify punt-labs/lux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 73.3% | Not in cards |
| Vals Smith: schubydoo/clauster Agents modify schubydoo/clauster to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 96.7% | Not in cards |
| Vals Smith: tensorflow/tensorflow Agents modify tensorflow/tensorflow to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 90% | Not in cards |
| Vals Smith: vastsa/PI-Desktop Agents modify vastsa/PI-Desktop to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | | Not in cards |
| Vals Smith: vllm-project/llm-compressor Agents modify vllm-project/llm-compressor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Sep 2026 | 96.7% | Not in cards |
| VAmoS Pro Bench Voice agents handle several utility billing requests in one call. A model checks the spoken amounts and recorded actions against the requirements for each request. | University | Sep 2026 | | Not in cards |
| Vibe Code Bench 1-100 Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it. | Expert | Sep 2026 | | Not in cards |
| Voice Arena Goal Dataset Judges compare two calls and choose which caller achieved the airline task more successfully. Human preferences provide the reference. | Everyday | Sep 2026 | | Not in cards |
| WeirdML v3 Agents solve 11 hand-made machine learning and data analysis tasks on a GPU with unfamiliar data and hidden test answers. Scored by token-weighted best results. | Expert | Sep 2026 | 43.1% | Not in cards |
| WitnessBench Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels. | Expert | Sep 2026 | 46.5% | Not in cards |
| WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. | Everyday | Sep 2026 | 42.3% | Not in cards |
| WRENCH (Docker Compose) Agents maintain a service pipeline through seeded disruptions. The grader measures retained throughput and recovery. | University | Sep 2026 | 0.81 | Not in cards |
| XYEval Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice. | Expert | Sep 2026 | 60.9% | Not in cards |
| ZenML Bench Agents build, repair, and investigate machine learning pipelines in ZenML. Eighteen tasks grade the state the framework records. | Expert | Sep 2026 | Not in cards | |
| τ^τ-bench Coding agent builds a customer-service agent from business records, a client and an API. The built agent is scored against held-out simulated users. | Expert | Sep 2026 | 23.9% | Not in cards |
| AA-AnalystAgent Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right. | Expert | Aug 2026 | 62.5% | In use |
| Active-SWE Coding agents inspect repository snapshots and fix bugs without issue reports. Evaluation checks patches against reference tests and measures localization precision and recall. A separate stage uses generated tests and model review to assess newly revealed bugs. | Expert | Aug 2026 | 20% | Not in cards |
| ADeptS-Bench Desktop Agents choose safe interface actions on the Desktop test set. The published safety score combines task success and attack resistance. | Expert | Aug 2026 | | Not in cards |
| Agent-Only Fantasy Football League Models draft NFL players, set lineups, and negotiate trades using the same prompt and tools. The organizer compares points, lineup efficiency, and cost per point. | Everyday | Aug 2026 | 1371.4 | Not in cards |
| Alibaba CTF Agents solve 87 capture-the-flag tasks in Alibaba’s collection of competition challenges from 2023 through 2026. | Expert | Aug 2026 | Not in cards | |
| Apodex Discovery Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors. | Research frontier | Aug 2026 | 2.80 | Not in cards |
| AQuA Two LLM research loops (factor discovery, model development) reuse evidence from their own past experiments inside sealed sandboxes. Scored by IC and held-out Sharpe. | Research frontier | Aug 2026 | Not in cards | |
| Artificial Analysis Coding Agent Index v1.4 Version 1.4 of Artificial Analysis's coding agent index, which moved to the full Terminal-Bench 2.1 and began scoring reward-hacked runs as zero. | Aug 2026 | 67.0 | In use | |
| Artificial Analysis Search Index One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores. | Expert | Aug 2026 | Not in cards | |
| ASI-Bench AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge. | Research frontier | Aug 2026 | 55.8 | Not in cards |
| Aspire Agent gets only a vague capability goal and must choose data, training and self-checks to evolve weights or harness. Scored on 520 hidden items. | Research frontier | Aug 2026 | | Not in cards |
| AutoDS scientific machine learning cases Agents build predictive models for maize yield, polymer glass transition temperature, and supercomputer job outcomes. Each case supplies a train split and a held-out test. | Expert | Aug 2026 | Not in cards | |
| Balatro Bench (Michael Andrzejewski) Agents play one fixed Balatro seed with the Red Deck on White Stake. The maker ranks runs by ante reached, breaking ties by best hand. | Everyday | Aug 2026 | 18.0 | Not in cards |
| Behavior2Code Agents reconstruct command line programs from documentation and observed behavior. Hidden tests compare each implementation with the original. | Expert | Aug 2026 | 31.7% | Not in cards |
| BixBench3 Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts. | Research frontier | Aug 2026 | | Not in cards |
| Blobfish Litigation Bench Agents complete 100 tasks in a simulated litigation practice. The grader compares the changes agents make to case records against required outcomes. | Expert | Aug 2026 | Not in cards | |
| BoardBench 0.1 Models play Yatzi, Scotland Yard, and Splendor. Game engines identify the best action at each decision and the scorer counts exact matches. | Everyday | Aug 2026 | 91.4% | Not in cards |
| BrowseComp Extended Agents browse the web to find hard-to-locate facts with short answers. Mercor adds 100 new expert-written tasks to the 130 public ones and reports both scores. | Expert | Aug 2026 | 73% | Not in cards |
| Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. | Expert | Aug 2026 | | System cards only |
| ClawProBench Agents on stateful runtimes are scored with their traces, covering evidence gathering, runtime routing, safety boundaries and repeated runs in frozen workplace setups. | Expert | Aug 2026 | 80% | Not in cards |
| CliniCARE-Bench Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases. | Expert | Aug 2026 | 76.1% | Not in cards |
| CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. | Expert | Aug 2026 | Not in cards | |
| ContinualSkillBench Agents work through five domains of 100 linked subtasks of rising difficulty, testing whether skill libraries they build in context actually help later tasks. | Expert | Aug 2026 | 64.1% | Not in cards |
| CorporateBench Agents answer questions over the email archives of four synthetic companies, up to 232,692 documents. Answers are checked against each company's ground-truth knowledge graph. | Expert | Aug 2026 | 82.2% | Not in cards |
| CounselBench-100 Agents investigate legal matters and decide which actions the evidence supports. The verifier checks decisions, changes to case records, and advice across 100 synthetic tasks. | Expert | Aug 2026 | Not in cards | |
| Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. | Expert | Aug 2026 | | System cards only |
| CyberStrike Agents complete cybersecurity tasks. Automated checks grade the outcomes. | Expert | Aug 2026 | 31.5% | Not in cards |
| data-eng-bench Agents build and repair dbt models across 103 retail warehouse tasks. The verifier compares materialized tables row by row against a reference solution. | University | Aug 2026 | Not in cards | |
| DevOpsBench-100 Agents resolve 100 operations and reliability tasks in a simulated online store. The verifier compares system state before and after their work. | Expert | Aug 2026 | Not in cards | |
| dfbench Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately. | Expert | Aug 2026 | 69% | Not in cards |
| DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. | Everyday | Aug 2026 | 71.4% | Not in cards |
| Dirty Integration Agents collect every record from a paginated API despite unreliable cursors, rate limits, changing field types and timeouts. | University | Aug 2026 | Not in cards | |
| DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring. | Expert | Aug 2026 | | In use |
| DuMateBench Agents complete real-world productivity and office-work tasks in sandboxed workspaces with tools. Task evaluators check the requested artifact and record a reward. | Expert | Aug 2026 | Not in cards | |
| E-Bench-Code (Tencent) Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published. | Expert | Aug 2026 | 83.3% | In use |
| E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. | University | Aug 2026 | 14.3 | Not in cards |
| EarthVerse Agents investigate documented natural hazards using event data from different sources. The evaluator checks individual claims and the calculations that support them across 405 tasks. | Expert | Aug 2026 | 85% | Not in cards |
| EduClaw-Bench Agents tutor a simulated learner over 30 days. The benchmark tracks learning gains, responsiveness, helpfulness, and curriculum design across 55 scenarios. | University | Aug 2026 | Not in cards | |
| EmbodiedCLUE-VLA RoboTwin Evaluation Robot policies perform ten two-handed manipulation tasks under randomized RoboTwin 2.0 Hard conditions. The score averages task success rates over twenty episodes per task. | Everyday | Aug 2026 | Not in cards | |
| Enterprise Task Pareto Frontier (Glean) Models complete 1,000 workplace tasks. A model compares paired answers for correctness, task completion, and readiness for use. Glean estimates preference ratings and cost per task. | University | Aug 2026 | Not in cards | |
| FactoryBench-100 Agents complete 100 manufacturing workflows through simulated ERP, email, spreadsheet, and chat tools. FactoryScore checks their decisions and the state left in each system. | Expert | Aug 2026 | 76.2% | Not in cards |
| FinDeepIndicator Agents construct financial indicators by specifying formulas, retrieving market data, and calculating results. The evaluation checks performance at each step. | University | Aug 2026 | | Not in cards |
| Frontier Agentic ML & DS Agents solve machine learning and data science tasks in a terminal. Verifiers check their deliverables and predictive quality. | Expert | Aug 2026 | 64.6% | Not in cards |
| Frontier Agentic STEM Agents produce scientific results in a terminal. Task verifiers recompute the expected results from the underlying data. | Expert | Aug 2026 | 55.1% | Not in cards |
| FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. | Expert | Aug 2026 | 20.6% | Not in cards |
| FrontierPhysics Agents do end-to-end physics research on 56 tasks that 42 physicists from 26 institutions built from their own work, across nine domains of theoretical, computational and experimental physics. Each task asks the agent to review the literature, plan, run the research and write a manuscript. Deterministic verifiers and expert rubrics grade whether the research is correct and publishable, over 3 trials. | Expert | Aug 2026 | 27.4% | Not in cards |
| Game Arena Nine Men's Morris Language models play Nine Men's Morris. Forming a row of three, called a mill, lets a player remove one of the opponent's pieces. | Everyday | Aug 2026 | | Not in cards |
| Game Arena Reversi Language models play Reversi, also called Othello, flipping the opponent's disks that they flank in a line. | Everyday | Aug 2026 | | Not in cards |
| Game Arena Word Art Language models take turns drawing a word as ASCII art and guessing what the other model drew. | Everyday | Aug 2026 | | Not in cards |
| GameXpert-Bench Agents build games from 97 briefs, repair bugs in 100 tasks, and improve 17 games over six requests each. Each track reports its own scores. | University | Aug 2026 | 70.9 | Not in cards |
| GDPevo Agents learn from training tasks drawn from enterprise workflows and must reuse that experience on related test tasks. Gains at test time measure self-evolution. | Expert | Aug 2026 | 67.1% | Not in cards |
| Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| Gray Swan Cyber Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on cyber tasks built from Google scenarios. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. | Expert | Aug 2026 | 10.1% | Not in cards |
| HarnessOpt-Bench Coding agents edit a target agent’s harness under a fixed evaluation budget. The grader measures improvement over the starting harness on held-out tasks. | Expert | Aug 2026 | 0.47 | Not in cards |
| HEPToolBench 1.2 Models generate inputs for particle physics simulations and inspect the resulting files. The main suite has 28 tasks and a separate extension adds three debugging tasks. | Expert | Aug 2026 | | Not in cards |
| HM3D-MFMON Agents navigate across floors and find a sequence of objects. | Everyday | Aug 2026 | Not in cards | |
| HWE-bench CAE Agents run engineering simulations, extract numerical results, and report them. A verifier repeats the extraction from the solver output and checks engineering tolerances. | Expert | Aug 2026 | 0.90% | Not in cards |
| Hy-BrowseComp-Pro2 Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-LifeSearch Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| HybridDeepResearch Agents combine database queries and web search to answer research questions under constraints. | Expert | Aug 2026 | | Not in cards |
| HyDRA-Bench Agents interleave database queries and document retrieval to answer questions. A model judge checks the final answer. | University | Aug 2026 | | Not in cards |
| InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. | Expert | Aug 2026 | 46.9% | System cards only |
| Insight Consolidation v1 Agents read customer account documents, infer why the customer acted, and cite supporting records. The evaluator combines a model judgment with citation accuracy. | University | Aug 2026 | Not in cards | |
| IronClad Agents repair security misconfigurations in Terraform repositories. The grader verifies that the repaired configuration satisfies policy and preserves the intended infrastructure. | University | Aug 2026 | Not in cards | |
| ITSMBench Agents operate an IT service desk across 53 conversations with simulated colleagues. The verifier grades the resulting database state, including permissions and required actions. | University | Aug 2026 | 0.70 | Not in cards |
| KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. | Expert | Aug 2026 | 37.2% | System cards only |
| LAB: Law Firm Knowledge Agents search a simulated law firm’s matter files and deliver work that satisfies legal task criteria. | Expert | Aug 2026 | Not in cards | |
| LedgerBench-100 Agents perform 100 corporate finance tasks through 66 tools in a shared enterprise system. SQL checks grade the changes they make. | Expert | Aug 2026 | Not in cards | |
| LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. | Everyday | Aug 2026 | | Not in cards |
| LLM Fantasy Football Arena Classic Models manage two NFL fantasy teams each. One league allows trades and messages. The other disables both. Each league scores its own draft and season. | Everyday | Aug 2026 | | Not in cards |
| LoopArena Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit. | Expert | Aug 2026 | 36.3% | Not in cards |
| MineAmongUs Vision-language agents play an embodied Among Us game with verbal and nonverbal deception. | Everyday | Aug 2026 | | Not in cards |
| MIVAS Bench Voice agents handle calls for a dermatology practice, a law firm, and a retailer. Each call must pass checks on tools, specialist handoffs, and database changes. | University | Aug 2026 | 56.5% | Not in cards |
| MobilePA-Bench Agents complete 1,705 mobile planning tasks using tools, persistent memory, reusable skills, and specialist agents. Task-specific checks evaluate tool calls, state changes, and interaction behavior. Additional gates check required memory access and skill loading. | Everyday | Aug 2026 | 76.9% | Not in cards |
| MOMENT Eval Models use a calculator and Python to solve 91 structural engineering problems. They return a value or classification that a program checks against the answer key. | Expert | Aug 2026 | 95.6% | Not in cards |
| NanoGPT Speedrun Frontier Agents change a GPT optimizer on fixed GPUs. Each candidate must reach the target validation loss across eight fixed seeds using fewer training steps. | Research frontier | Aug 2026 | 2726.0 | Not in cards |
| NCP-Bench Models narrate interactive stories while preserving facts, commitments, and plot order against player interventions. A model auditor checks 100 movie-derived environments after each turn. The evaluation reports survival, conflicts, and narrative progress. | Everyday | Aug 2026 | 32.9 | Not in cards |
| OfficeQA Pro V2 Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match. | University | Aug 2026 | 54.4% | In use |
| OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. | University | Aug 2026 | | Not in cards |
| Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. | University | Aug 2026 | | System cards only |
| OmniMech Models write CAD programs to reconstruct parts from dimensioned drawings. The evaluator compares the reconstructed geometry with reference parts and measures relative volume error. | University | Aug 2026 | 15.9% | Not in cards |
| OpenStudio-MCP LLM Test Benchmark v1.2.1 Agents change building energy models with OpenStudio tools. The grader checks the tool calls and inspects the resulting model against task requirements. | University | Aug 2026 | 100% | Not in cards |
| OSWorkerBench Desktop agents complete long-horizon office tasks across business applications with executable evaluators and optional demonstrations. | Expert | Aug 2026 | Not in cards | |
| OSWorld 2.0 (Aug 2026) Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. | University | Aug 2026 | 77.9% | System cards only |
| Palmyra X6 Enterprise Workflow Evaluations Agents perform production workflows that require retrieval, tool use, content generation, or delegation. Writer compares capability and cost. | Expert | Aug 2026 | Not in cards | |
| PlayWorld An agent player pursues objectives inside video world models. Gemini 3.1 Pro applies weighted rubrics for geometry, interaction, and state evolution across the resulting videos. | Everyday | Aug 2026 | Not in cards | |
| PosterBench Agents turn research papers into posters. Programmatic checks and visual model judgments score seven content and design dimensions, with ceilings for serious rendering or layout failures. | University | Aug 2026 | Not in cards | |
| PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. | Expert | Aug 2026 | 63.3% | Not in cards |
| QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. | University | Aug 2026 | | In use |
| Retrobench v0.4 Agents reconstruct C from five Nintendo 64 assembly targets. The organizer scores the average percentage of matching bytes under a budget of 25 compilations per target. | Expert | Aug 2026 | | Not in cards |
| RevBench (Polyvor Labs) Agents analyze supplied artifacts in 22 local reverse engineering puzzles. The verifier checks each submitted answer for an exact match. | Expert | Aug 2026 | 50% | Not in cards |
| RSI Bench Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks. | Research frontier | Aug 2026 | 38.9 | Not in cards |
| RSI-Exam Agents get a weak method on a long research task and must turn it into one that scores better on hidden data. It measures recursive self-improvement on executable research. | Research frontier | Aug 2026 | 53.6% | Not in cards |
| S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. | Expert | Aug 2026 | Not in cards | |
| SalesBench-100 Agents reconcile sales evidence, update authorized records, and deliver executive briefs across 100 workflows. Checks grade procedure, system state, and outputs. | Expert | Aug 2026 | Not in cards | |
| SecIT Bench Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions. | Expert | Aug 2026 | | Not in cards |
| SemaPLC Project-Context Track Agents implement plant control logic inside existing Structured Text projects. The evaluator measures compilation, static checks, and agreement with reference outputs during execution. | Expert | Aug 2026 | 100% | Not in cards |
| Small Model Agent Bench Models choose tools and supply their arguments in 28 Chinese and English episodes. Fixed tool responses let the grader check planning, recovery, and task completion. | School | Aug 2026 | | Not in cards |
| Speech Agent Arena People hold live conversations with two hidden speech-to-speech agents on the same scenario, some with tool calls, then pick the one they preferred. Votes are fit to an Elo rating, and the arena also tracks task success. | Everyday | Aug 2026 | 1,070 | In use |
| SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. | Expert | Aug 2026 | 56.9% | In use |
| Sregym env Agents diagnose incidents in a running service with generated logs and traffic. The grader checks service recovery, the underlying fault and collateral damage. | University | Aug 2026 | Not in cards | |
| SWE Atlas Codebase QnA Extended Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores. | Expert | Aug 2026 | 44.7% | Not in cards |
| SWE Refactor Bench Agents migrate 20 whole open-source repositories to a new language, framework, platform, or build system. Graded by migration audit, behavioral tests, and adversarial verifier models. | Research frontier | Aug 2026 | 47% | In use |
| SWE-Bench ProMax Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions. | Expert | Aug 2026 | 41.2% | Not in cards |
| SWE-bench Science Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix. | Expert | Aug 2026 | 50.4% | Not in cards |
| SWE-bench Verified Extended Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores. | Expert | Aug 2026 | | Not in cards |
| SWE-Marathon-Ext Agents build 12 software products from specifications. Automated tests and browser checks grade the applications. | Expert | Aug 2026 | 74% | Not in cards |
| Terminal-Bench 2.1 Extended Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores. | Expert | Aug 2026 | | Not in cards |
| Terminal-Bench 4.0 Agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. | Expert | Aug 2026 | 64.8% | In use |
| Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. | Expert | Aug 2026 | 68.1% | In use |
| Thinkingbox-bench Agents complete 507 business workflows through tools and simulated user conversations. Executable checks require correct backend updates without forbidden side effects. | University | Aug 2026 | 66.5% | Not in cards |
| Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. | Expert | Aug 2026 | | Not in cards |
| TxBench-Antibody-Discovery Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision. | Expert | Aug 2026 | 67.7% | Not in cards |
| Vals Index v2 Version 2 of Vals AI's GDP-weighted index, which brought back a legal sector and added financial modeling in Excel and code migration. | Aug 2026 | 69.7% | Not in cards | |
| Vals RSI Index Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness. | Research frontier | Aug 2026 | 0.38 | Not in cards |
| Vals Smith: abhishek203/SfAgent Agents modify abhishek203/SfAgent to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 80% | Not in cards |
| Vals Smith: apache/spark Agents modify apache/spark to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 84.5% | Not in cards |
| Vals Smith: arandomhooman/hoomans-morphe-patches Agents modify arandomhooman/hoomans-morphe-patches to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 20% | Not in cards |
| Vals Smith: arduino/arduino-ide Agents modify arduino/arduino-ide to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 43.3% | Not in cards |
| Vals Smith: astral-sh/uv Agents modify astral-sh/uv to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 66% | Not in cards |
| Vals Smith: benredmond/apex Agents modify benredmond/apex to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 63.3% | Not in cards |
| Vals Smith: BerriAI/litellm Agents modify BerriAI/litellm to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | | Not in cards |
| Vals Smith: chrisanderton/tyk-gateway Agents modify chrisanderton/tyk-gateway to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 76.7% | Not in cards |
| Vals Smith: conjfrnk/blackbird Agents modify conjfrnk/blackbird to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 56.7% | Not in cards |
| Vals Smith: conjfrnk/memovox Agents modify conjfrnk/memovox to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 90% | Not in cards |
| Vals Smith: flutter/flutter Agents modify flutter/flutter to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | | Not in cards |
| Vals Smith: geopandas/geopandas Agents modify geopandas/geopandas to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | | Not in cards |
| Vals Smith: goharbor/harbor Agents modify goharbor/harbor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 60% | Not in cards |
| Vals Smith: jax-ml/jax Agents modify jax-ml/jax to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 79.6% | Not in cards |
| Vals Smith: ladydascalie/core Agents modify ladydascalie/core to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | | Not in cards |
| Vals Smith: leanprover-community/mathlib4 Agents modify leanprover-community/mathlib4 to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 76.7% | Not in cards |
| Vals Smith: mirumee/ariadne Agents modify mirumee/ariadne to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 83.3% | Not in cards |
| Vals Smith: nationalsecurityagency/ghidra Agents modify nationalsecurityagency/ghidra to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 73.3% | Not in cards |
| Vals Smith: NVIDIA/nccl Agents modify NVIDIA/nccl to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 46.7% | Not in cards |
| Vals Smith: oven-sh/bun Agents modify oven-sh/bun to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 54.2% | Not in cards |
| Vals Smith: palantir/blueprint Agents modify palantir/blueprint to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 93.2% | Not in cards |
| Vals Smith: poalrom/sharepaste Agents modify poalrom/sharepaste to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 50% | Not in cards |
| Vals Smith: sashabaranov/go-openai Agents modify sashabaranov/go-openai to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 96.7% | Not in cards |
| Vals Smith: sgl-project/sglang Agents modify sgl-project/sglang to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 78% | Not in cards |
| Vals Smith: skypilot-org/skypilot Agents modify skypilot-org/skypilot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 30% | Not in cards |
| Vals Smith: suitedaces/dorabot Agents modify suitedaces/dorabot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | | Not in cards |
| Vals Smith: tock/tock Agents modify tock/tock to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 86.7% | Not in cards |
| Vals Smith: torvalds/linux Agents modify torvalds/linux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 74.1% | Not in cards |
| Vals Smith: vercel/next.js Agents modify vercel/next.js to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Expert | Aug 2026 | 74.3% | Not in cards |
| VideoGAIA Agents inspect selected video segments, search the web, and read pages before answering questions. Model judges check answers, and the harness reports accuracy by category and difficulty. | Everyday | Aug 2026 | | Not in cards |
| VWE-Bench Multimodal agents build interactive 3D worlds from a user request. They infer intent, plan the scene layout, call 3D tools for asset retrieval, editing and rendering, and revise from rendered feedback. The benchmark has 2,616 3D assets, 323 human-annotated seed worlds and 6,828 queries. Verified queries have ground truth, and the rest are scored with rubrics. | Expert | Aug 2026 | 57.3% | Not in cards |
| WildArtifactBench Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo. | Expert | Aug 2026 | Not in cards | |
| WRENCH (Factorio) Agents build Factorio factories and recover from unannounced disruptions. The evaluator measures how much production the factory retains after a disruption. | University | Aug 2026 | 1.49 | Not in cards |
| 24/7 ClawBench 2.0 Agents do always-on assistant work through OpenClaw. Tasks span several days, events arrive at the same time, and interruptions are routine. | Expert | Jul 2026 | 48.3% | System cards only |
| 3D SubwayBench Agents use Command Code to build a Subway Surfers style endless runner in a single HTML file. The scorecard grades six features from the submitted code. | University | Jul 2026 | 5.00 | Not in cards |
| AbstentionAct Agents handle HR and operations requests with missing information or authority. The benchmark checks whether agents clarify, abstain, or complete a permitted action. | Expert | Jul 2026 | Not in cards | |
| ActiveVision Models inspect images repeatedly to count regions, trace structures, and compare visual attributes. The released split has five examples for each of 17 tasks. Exact match checks the final answer after normalizing case, spacing, and separators. | Everyday | Jul 2026 | 50.6% | Not in cards |
| Agent Behavior Bench Agents are scored on process quality, including tool-use behavior, efficiency and discipline, together with task completion. | Expert | Jul 2026 | 65% | System cards only |
| Agent Memory Leaderboard Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework. | Everyday | Jul 2026 | Not in cards | |
| AgentHarm Verified (Meta) Agents receive 108 malicious AgentHarm tasks kept after human review, plus a benign set. Reports average harm score and false refusal rate. | Expert | Jul 2026 | | System cards only |
| Agentic Trader Agents search for market information and place simulated foreign exchange orders using live prices. Each agent manages its own positions, cash, and margin. | University | Jul 2026 | | Not in cards |
| Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. | Expert | Jul 2026 | 78.3% | System cards only |
| AgenticDataBench Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios. | Expert | Jul 2026 | 48.8% | Not in cards |
| AI Battle Arena Models play poker, Gomoku, and Connect Four through the same interface. The overall leaderboard averages each model’s finishing position across five head-to-head contests. | Everyday | Jul 2026 | 88.0 | Not in cards |
| AISE-Bench Agents answer academic information-seeking questions through multi-step knowledge-graph APIs and source-grounded responses. Planning, execution and answer-grounding metrics score each trajectory. | Expert | Jul 2026 | Not in cards | |
| APEX-Accounting Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria. | Expert | Jul 2026 | | In use |
| ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. | Everyday | Jul 2026 | 47.6% | Not in cards |
| Artificial Turf War Models bid for draft position, pick NFL players, and submit weekly lineups. The league ranks teams by wins and breaks ties with fantasy points. | Everyday | Jul 2026 | 6.00 | Not in cards |
| ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. | Everyday | Jul 2026 | System cards only | |
| AutoCAD Bench Computer-use agents turn a dimensioned reference drawing into a correct, editable DWG in AutoCAD through the screen alone, with mouse and keyboard. 50 tasks span 2D drafting and 3D modeling at four difficulty levels. | Expert | Jul 2026 | 46% | Not in cards |
| aws-bench Coding agents and model pairs diagnose misconfigurations, provision infrastructure and operate live cloud setups in disposable real AWS accounts. | Expert | Jul 2026 | Not in cards | |
| BackSearch Paid API that searches a frozen web archive as of a given date, for forecasting backtests and reproducible web-enabled evals. A tool, not a benchmark. | Jul 2026 | Not in cards | ||
| BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. | Everyday | Jul 2026 | Not in cards | |
| Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. | Expert | Jul 2026 | | In use |
| BioCircuit Agents identify groups of features in Qwen3-4B that control a behavior or restore a damaged output. The grader tests their interventions on held-out prompts. | Research frontier | Jul 2026 | Not in cards | |
| BioSecBench-Refusal Agents run 61 routine and 46 concealed-hazard biology analysis tasks. The score pairs red-team refusal with routine completion. | Expert | Jul 2026 | 62.4% | System cards only |
| BioSecBench-Surveillance Agents turn raw pathogen sequencing data into surveillance results. The endpoint pass rate excludes refused runs. | Expert | Jul 2026 | 60.8% | System cards only |
| Bot Analysis Arena Models write trading rules for simulated US stock accounts. Some agents revise their rules after trades. Software tests each revision against the current rules before adoption. | University | Jul 2026 | Not in cards | |
| Cekura Bench Models complete separately scored speech recognition, speech generation, voice-agent, real-call, turn-taking and voice-quality benchmarks. Cekura publishes no combined score. | Everyday | Jul 2026 | Not in cards | |
| Cekura Production Voice Agents Published composite score for production voice agents. | Everyday | Jul 2026 | Not in cards | |
| Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability. | Expert | Jul 2026 | 59.9% | System cards only |
| Command Code Flappy Bird Agents use Command Code to build a Flappy Bird game with Canvas. The scorecard grades features including collisions, window resizing, and offline play. | University | Jul 2026 | 4.83 | Not in cards |
| Command Code Pixel Art Space Shooter Agents use Command Code to build a scrolling Canvas game with cave walls and collectible gems. The scorecard grades features including collisions and offline play. | University | Jul 2026 | 4.83 | Not in cards |
| Composite-Bench Agents solve generated browser tasks in planning, rule extraction, deduplication, routing, and information gathering. Success requires matching a certified optimal solution. A separate number-partitioning task measures solution quality rather than contributing to the success rate. | Expert | Jul 2026 | 74% | Not in cards |
| Cost per Successful Task Agents solve 40 Terminal-Bench tasks. The benchmark divides spending across all attempts by the number of successful runs. | Expert | Jul 2026 | | Not in cards |
| CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. | Research frontier | Jul 2026 | 34.2% | Not in cards |
| CursorBench 3.2 Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.2 added instruction following and advanced tool use problems. Agentic graders score correctness. | Expert | Jul 2026 | 73.4% | In use |
| CyberGym (final-submission metric) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. | Expert | Jul 2026 | | In use |
| Cybersecurity and Privacy Legal Agent Eval Agents write legal risk memos for five synthetic privacy and incident reporting matters. The grader checks the completed memo and its coverage of legal issues. | Expert | Jul 2026 | Not in cards | |
| Deep Research Bench (Moonshot) Agents answer deep-research queries curated by domain experts. Expert-aligned rubrics grade the reports. | Expert | Jul 2026 | 90% | System cards only |
| DeepsecBench Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings. | Expert | Jul 2026 | 61.2 | Not in cards |
| DexVerse Robot policies perform tabletop manipulation tasks in Isaac Lab, including tool use, bimanual coordination, and sequences with several goals. Task-specific success conditions check completion under different visual settings and robot embodiments. | Everyday | Jul 2026 | 34% | Not in cards |
| DiligenceBench 150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness. | Expert | Jul 2026 | | In use |
| Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. | Expert | Jul 2026 | 97.3% | Not in cards |
| E-Bench (Tencent) Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published. | Expert | Jul 2026 | 80.6% | In use |
| EBR-bench Agent plays the board game Earthborne Rangers 10-30 times in a row. Scored on objectives completed (of 21) to test learning from experience. | Everyday | Jul 2026 | 92.4% | Not in cards |
| EdgeBench Agents iterate for up to 12 hours on 134 executable real-world tasks with multi-level feedback. Scores are tracked at 2 hour steps to measure learning from the environment. | Expert | Jul 2026 | 51.3 | Not in cards |
| Employment Law Legal Agent Eval Agents review employment case files and write risk memos. The grader checks whether the memos identify planted legal issues and avoid unsupported findings. | Expert | Jul 2026 | Not in cards | |
| Enterprise Vendor Eval Agents review vendor documents, question an internal IT team, and recommend whether to approve a purchase. The grader checks their decision and identified requirements. | University | Jul 2026 | Not in cards | |
| Enterprise-Bench L1-L2 Agents answer 14 operational questions across connected enterprise records. Tasks require either retrieving facts or combining evidence under business rules. | University | Jul 2026 | 82.9% | Not in cards |
| EvoPolicyGym Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves. | Expert | Jul 2026 | 0.89 | Not in cards |
| Excel Modeling Benchmark Agents build investment banking and private equity financial models in Excel from a written brief and source files. Rubrics give partial credit for tabs, structure, formulas and formatting. | Expert | Jul 2026 | 76.7% | Not in cards |
| FACTS Search V2 Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle. | University | Jul 2026 | 89.9% | Not in cards |
| FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. | Jul 2026 | 84% | System cards only | |
| Feishu Office v1 Agents handle 140 Chinese workplace requests in a simulated Feishu workspace. The grader checks the resulting calendars, documents, tables, and approvals. | University | Jul 2026 | Not in cards | |
| Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. | Expert | Jul 2026 | 62.6% | System cards only |
| Founder Bench Agents run four matched businesses per model through the acoco platform. The evaluation tracks real advertising, customer attention, and website traffic after seven days. Its headline customer-impressions metric adds ad impressions to organic site visits. | Expert | Jul 2026 | 127902.0 | Not in cards |
| FrontierCode 1.1 Real-repo coding tasks graded for mergeability: correctness, regressions, lint, tests, scope and code quality. Failing any blocker scores zero. | Expert | Jul 2026 | 53.5% | In use |
| FrontierFinance Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics. | Expert | Jul 2026 | 55.9% | In use |
| Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. | Everyday | Jul 2026 | Not in cards | |
| Game Arena Bargaining Two language models negotiate how to split a pool of items, each with private item values. Each side tries to reach a good deal without giving away its values. | Everyday | Jul 2026 | | Not in cards |
| Game Arena Coin Game Two language models move on a grid and pick up coins. Each should collect the coins that score for it and avoid the rest. | Everyday | Jul 2026 | 514 | Not in cards |
| Game Arena Lines of Action Language models play Lines of Action, where a piece moves as far as the number of pieces on its line. The goal is to join all of one's pieces into one group. | Everyday | Jul 2026 | | Not in cards |
| Game Arena Ultimate Tic-Tac-Toe Language models play nine linked tic-tac-toe boards. Each move decides which small board the opponent must play next. | Everyday | Jul 2026 | | Not in cards |
| GameEngineBench Agents edit C++ files in existing Unreal Engine 5 projects. Runtime tests check the changes, and a model audits whether the behavior meets the specification. | University | Jul 2026 | 55.5% | Not in cards |
| GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. | Everyday | Jul 2026 | Not in cards | |
| Gauntlet Agents develop game controllers through practice. The authors freeze each controller and test it on unseen games without further model calls. | University | Jul 2026 | Not in cards | |
| Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. | Everyday | Jul 2026 | Not in cards | |
| Gorgias AI Agent Competitive Benchmark Agents recommend products and answer support questions on live Shopify stores. Gorgias combines answer quality, response time, and the share of conversations handled without human help. | Everyday | Jul 2026 | Not in cards | |
| HANDBOOK.md Agents do routine company work with mock email, chat, calendar, issue and commerce tools while following a 20 to 124 page handbook. Programmatic criteria check required and forbidden actions, and a trial passes only if all pass. | University | Jul 2026 | 52.3% | In use |
| HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. | Everyday | Jul 2026 | | Not in cards |
| Hy-Backend 2.0 Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Hy-CompanyBench Coding agents complete Tencent in-house company engineering tasks. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Hy-Euler pro Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Hy-FinModelBench Agents build financial models for Tencent in-house finance tasks. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Hy-SkillsWorld Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. | Expert | Jul 2026 | System cards only | |
| InfraBench Agents deploy, profile, debug, and optimize model workloads on AMD MI35X GPUs. Task-specific verifiers grade 20 engineering assignments. | Expert | Jul 2026 | Not in cards | |
| KAET (Kimi Autonomous Execution Tasks) Agents execute long-horizon tasks that simulate real user requests and enterprise system operations. | Expert | Jul 2026 | 83.5% | System cards only |
| KAT Claw Bench Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables. | University | Jul 2026 | | In use |
| KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages. | University | Jul 2026 | | In use |
| KernelBench (kernelbench.com) Coding agents write GPU kernels for real model workloads, such as fused MoE layers, paged attention and FP8 matrix multiplies. The site groups tasks into suites: Mega (one decode task), CUDA (4 tasks) and Hard (6 tasks). A run counts only if it passes an audit of correctness. Scores are speedup over a baseline or percent of the hardware roofline. | Expert | Jul 2026 | | Not in cards |
| Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. | Expert | Jul 2026 | 38.9% | System cards only |
| Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. | Expert | Jul 2026 | System cards only | |
| Ko-GDPval Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format. | Expert | Jul 2026 | | In use |
| LAB: Diligence Agent researches a synthetic M&A virtual data room (thousands of documents, up to ~80M tokens) and drafts a diligence memo scored against planted-issue rubric criteria. | Expert | Jul 2026 | Not in cards | |
| Lamina Product Bench Agents build four small applications under three instructions: code directly, plan first, or use Lamina skills. A model judge grades each submission. | University | Jul 2026 | Not in cards | |
| LaminaBench v1 Agents implement the same product brief with and without Lamina skills. The package README specifies ten briefs across five categories and 20 paired Harbor tasks. | University | Jul 2026 | Not in cards | |
| LMArcade Agents build browser games from the same prompt. Players try two games and vote before learning which models and coding tools made them. | University | Jul 2026 | 1,798 | Not in cards |
| Long-Horizon-Terminal-Bench (LHTB) Terminal-Bench-style container tasks needing hundreds of steps. Subtasks graded deterministically for dense partial-credit reward. Solved at reward >=0.95. | Expert | Jul 2026 | | Not in cards |
| LoopsBench Agents complete 112 software tasks with dependencies among smaller development units. Executable tests measure resolved tasks and the fraction of units that pass. | Expert | Jul 2026 | 25% | Not in cards |
| Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. | Everyday | Jul 2026 | 3.23 | Not in cards |
| MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. | Expert | Jul 2026 | | Not in cards |
| Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. | Expert | Jul 2026 | | System cards only |
| MIRA Bench Agents carry out long multi-role enterprise tasks across several systems and decide when to delegate to subagents. | Expert | Jul 2026 | 64.1% | System cards only |
| MM-ToolSandbox Visual tool-calling agents complete 258 multi-image, multi-turn scenarios across 16 application domains and more than 500 tools. Rubric and entity-diff checks score the runs. | Expert | Jul 2026 | Not in cards | |
| Multi-Agent Arena Risk Four commanders, people or anonymized models, trade sealed messages and then attack for territory on a 42-territory map. Models are ranked by their mean share of territory, where 25 is an even split. | Everyday | Jul 2026 | 38.8% | Not in cards |
| Multi-Agent Arena Social Poker No-limit hold’em tournaments with table talk, where people and anonymized models share a table. Models are ranked by a rating fit on who finished ahead at each table. | Everyday | Jul 2026 | 1,553 | Not in cards |
| NanoGPT Expenditure Horizon Agents optimize NanoGPT training. METR estimates the budget at which human contributors become more cost effective than the agents. | Expert | Jul 2026 | Not in cards | |
| Olam Labs Multi-Agent Arena People and anonymized models share a table in social strategy games: Diplomacy, Social Poker, Codenames and Risk. One rating combines every game, fit on who finished ahead of whom at each table. | Everyday | Jul 2026 | 1,544 | Not in cards |
| OmniRouting Models route copper traces between pads on printed circuit boards. The evaluator counts required nets that connect without design rule violations. | University | Jul 2026 | 28% | Not in cards |
| Online Experience (Moonshot) Agents produce the deliverable file types that real online users request most often, following the distribution of real usage. | Everyday | Jul 2026 | 77.9% | System cards only |
| OptimBench Dispatch Agents route vehicles under capacity and time constraints, then revise their routes after a disruption. | University | Jul 2026 | 0.92 | Not in cards |
| ORCA-bench Agents investigate simulated production incidents and identify their root causes. The benchmark contains 1,079 tasks, including 324 held-out tasks covering 31 incidents. | Expert | Jul 2026 | 29.6% | Not in cards |
| ParliamentBench Models play Secret Hitler while deceiving, persuading, and reasoning under hidden roles. | Everyday | Jul 2026 | 81% | Not in cards |
| PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. | Expert | Jul 2026 | 4.25 | Not in cards |
| PCBWorld-Bench Agents route printed circuit boards in KiCad. A board earns a clean pass when all required connections are complete and KiCad reports zero design-rule errors. | University | Jul 2026 | 0.54 | Not in cards |
| PMPP-Hard Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. | Research frontier | Jul 2026 | 0.71 | In use |
| PostTrainBench 1.1 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Research frontier | Jul 2026 | 41.8% | In use |
| ProdBench (Tencent) Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3. | Expert | Jul 2026 | | In use |
| Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. | Expert | Jul 2026 | 99.8% | System cards only |
| ReactBench Coding agents do 39 realistic React tasks in two tracks. Write React tasks come from merged pull requests and are checked by hidden behavior tests. Fix React tasks ask the agent to remove React performance, accessibility and quality issues. A task passes only if the tests pass and a pinned React Doctor scan finds no new graded issue. | Expert | Jul 2026 | 54.4% | Not in cards |
| RH SWE-bench v1 Agents repair software across Red Hat ecosystem repositories and test each change against the issue requirements. | Expert | Jul 2026 | 63.3% | Not in cards |
| RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. | Everyday | Jul 2026 | 27.6% | In use |
| RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. | Research frontier | Jul 2026 | Not in cards | |
| RSIBench-Data Coding agents act as data-centric researchers. With a fixed Tinker post-training stack and budget, they refine training data from dev-benchmark feedback, and the final model is scored on held-out test benchmarks. | Research frontier | Jul 2026 | 32.6% | Not in cards |
| SecRespond Agents investigate compromised-host snapshots and write incident analyses and remediation plans. A model evaluator applies expert checklists separately to detection and response planning. | Expert | Jul 2026 | 79% | Not in cards |
| Silent Exit Models fix four typos without changing correct lines. A held-out test measures whether models learned to quit early to avoid penalties. | Everyday | Jul 2026 | Not in cards | |
| SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. | Expert | Jul 2026 | 61.1% | System cards only |
| SpecDecode RL Agents choose how many tokens to draft during speculative decoding. An offline grader measures speedup over the target model while requiring identical output. | Expert | Jul 2026 | Not in cards | |
| Speko Speech-to-Speech Benchmark (10-turn call) Voice agents answer a weather request with a tool, handle an interruption, and recall an earlier detail. A language model grades task success. | Everyday | Jul 2026 | | Not in cards |
| Speko Speech-to-Speech Benchmark (20-turn concierge call) Voice agents try to book a flight, handle a failed booking, and remember a changed date. Eight automated checks score their actions and replies. | Everyday | Jul 2026 | 1.00 | Not in cards |
| Spider 2.0-AIFunc Agents generate SQL that uses AI functions to solve 465 tasks across 125 databases. The evaluator compares execution results with reference results. | University | Jul 2026 | 70.3% | Not in cards |
| SREGym-Lite Agents diagnose and mitigate failures in running services on a Kubernetes cluster. | Expert | Jul 2026 | 81.3% | Not in cards |
| SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials. | Expert | Jul 2026 | | In use |
| Task AW IT support Agents complete five support workflows in simulated IT systems. Task checks cover onboarding, offboarding, and cleanup of accounts and assets. | University | Jul 2026 | Not in cards | |
| Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search. | Expert | Jul 2026 | 81.6% | Not in cards |
| Terminal-Bench 3.0 Agents complete 74 tasks across 7 domains in sandboxed terminal environments. Verifier scripts grade each trial as pass or fail. | Expert | Jul 2026 | 42.7% | In use |
| Terminal-Bench Data Agents solve terminal tasks in machine learning, data analysis, and scientific computing. Task verifiers grade the outputs. | Expert | Jul 2026 | 48.6% | Not in cards |
| Time Horizon Index: KSP Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions. | Expert | Jul 2026 | 91.3% | Not in cards |
| trivial-prompt-bench Models answer a short request through a terminal agent. Quesma adds the API charge to the estimated cost of the developer's waiting time. | Everyday | Jul 2026 | $0 | Not in cards |
| UAV Operator Models manage simulated drone missions as weather and flight restrictions change. The simulator scores their decisions from logged aircraft and mission state. | Expert | Jul 2026 | Not in cards | |
| UniClawBench Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill. | Everyday | Jul 2026 | 47.5% | In use |
| Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. | Expert | Jul 2026 | 57.5% | Not in cards |
| VAmoS Bench Voice agents answer simulated bank customers and use tools to manage credit cards. A model checks each call against its transcript and recorded actions. | University | Jul 2026 | | Not in cards |
| VariantBench Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result. | Expert | Jul 2026 | 52.8% | Not in cards |
| VerticalRTK Search and answer APIs answer questions about facts that change over time across seven real-world domains. An LLM judge compares answers with dated references. | Expert | Jul 2026 | Not in cards | |
| WANDR 500 wide-search tasks: find tens to hundreds of entities and facts, then return structured records with citations. | Everyday | Jul 2026 | 82.1% | In use |
| WorkBuddyBench-Office Agents edit office files and reconcile records across a workspace. One task requires updating an eleven-slide presentation from a spreadsheet while preserving unrelated slides. | University | Jul 2026 | 86.4% | Not in cards |
| WorkSurface-Bench Agents choose documents, tables, or graphs to answer workplace questions. The benchmark scores where agents look, what evidence they find, and their answers. | University | Jul 2026 | 10.3 | Not in cards |
| ZendoWorld Models infer hidden rules in the board game Zendo from labeled examples. | Everyday | Jul 2026 | Not in cards | |
| ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | | System cards only |
| ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | | System cards only |
| ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | 82.9% | System cards only |
| Act2Answer Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control. | Everyday | Jun 2026 | | Not in cards |
| AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. | Everyday | Jun 2026 | 67.8% | Not in cards |
| Agent Arena Arena ranks models as the orchestrator in real Agent Mode sessions on arena.ai. Scores use behavioral signals such as file downloads, disapproval events, retries and steerability. | Everyday | Jun 2026 | 28.6% | In use |
| Agent Startup Bench Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews. | Expert | Jun 2026 | | In use |
| Agentic Abstention Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on. | Everyday | Jun 2026 | Not in cards | |
| AgenticSTS Agents play the Silent in Slay the Spire 2 at its lowest difficulty. The authors compare wins with different memory settings and two other agents. | Everyday | Jun 2026 | | Not in cards |
| Agents' Last Exam 1,500+ expert-sourced tasks from real projects across 55 occupations in three difficulty tiers. Objective verifiers grade deliverables. 5-hour cap. | Expert | Jun 2026 | 38.2% | In use |
| Artificial Analysis Coding Agent Index v1.1 Version 1.1 of Artificial Analysis's coding agent index, which replaced SWE-Bench-Pro-Hard-AA with DeepSWE. | Jun 2026 | 80.0 | In use | |
| Artificial Analysis Intelligence Index v4.1 Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents. | Jun 2026 | 61.0 | In use | |
| ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. | Everyday | Jun 2026 | 48.2% | Not in cards |
| AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. | Expert | Jun 2026 | 0.68 | Not in cards |
| AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. | Expert | Jun 2026 | 66.5 | Not in cards |
| BabaIsBench Agents play levels of the puzzle game Baba Is You through a text interface and win by rewriting the rules. Models are compared on pass rate, speed and cost. | Expert | Jun 2026 | 100% | Not in cards |
| BigFinanceBench Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference. | Expert | Jun 2026 | 58.8% | In use |
| BIM-Edit Agents create, change, or delete elements in IFC building models. The evaluator checks the resulting geometry, attributes, and relationships. | University | Jun 2026 | | Not in cards |
| BTF-3 Agents forecast already-resolved binary and numeric questions while researching a frozen snapshot of the web. Scoring pools Brier and ranked probability scores. | Expert | Jun 2026 | 0.12 | Not in cards |
| BuildArena 2.0 Construction Challenge Agents build and test machines in Besiege for seasonal engineering challenges. The organizers score each entry using simulation results and the rules for its track. | Competition | Jun 2026 | Not in cards | |
| bun-server-bench Agents implement 50 backend services in Bun. Hidden tests grade behavior such as transaction handling, concurrent requests, and rate limits. | Expert | Jun 2026 | Not in cards | |
| CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. | Expert | Jun 2026 | $22,148,357 | Not in cards |
| ChainWorld Agents complete sequences of desktop tasks while preserving the state needed for later steps. OSWorld checks each step. | Everyday | Jun 2026 | 31% | Not in cards |
| Claw-SWE-Bench General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches. | Expert | Jun 2026 | | Not in cards |
| Code Migration Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. | Expert | Jun 2026 | 69.8% | Not in cards |
| CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. | University | Jun 2026 | Not in cards | |
| Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. | Expert | Jun 2026 | 0.20 | Not in cards |
| CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. | Everyday | Jun 2026 | | In use |
| Cua-Bench EDA (KiCad v1) Agents build or edit KiCad schematics through the desktop interface. The grader checks the resulting netlist and reports full passes, partial passes, and mean reward. | Expert | Jun 2026 | 24% | Not in cards |
| CyberBench (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Expert | Jun 2026 | 88.1% | Not in cards |
| CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. | Expert | Jun 2026 | 89% | Not in cards |
| CyberGym-E2E (615 tasks) Agents discover and demonstrate vulnerabilities on the older 615-task cohort. The headline is cumulative stage S3 success. | Expert | Jun 2026 | | Not in cards |
| DailyReport Search agents answer practical daily information requests. Evaluators assess the resulting reports across dimensions of user satisfaction. | Everyday | Jun 2026 | 2.89 | Not in cards |
| DeskCraft Desktop agents complete 538 Ubuntu tasks, including 152 tasks with later user clarifications or interruptions. Programmatic checks inspect final application state, saved files, and exported artifacts. | Expert | Jun 2026 | 29.8% | Not in cards |
| DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. | Expert | Jun 2026 | | Not in cards |
| DuetBench Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head. | Expert | Jun 2026 | 1,035 | Not in cards |
| DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. | Everyday | Jun 2026 | 10.6% | Not in cards |
| EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. | Everyday | Jun 2026 | Not in cards | |
| Eleusis Agents play cards to discover a hidden rule. The scorer tests whether a proposed rule makes the same decisions as the hidden rule. | Competition | Jun 2026 | 82.2% | Not in cards |
| EnterpriseClawBench Agents do 852 tasks rebuilt from real proprietary workplace agent sessions, each with fixtures, hard rules and semantic rubrics. | Expert | Jun 2026 | 66.3% | Not in cards |
| EpiBench Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result. | Expert | Jun 2026 | 39.6% | Not in cards |
| Evalatro v2 Models play Balatro from the game state and try to clear Ante 12. Illegal moves reduce the score for progress through the game. | Everyday | Jun 2026 | 46.1 | Not in cards |
| EvalConvoLearn Models simulate learners in tutoring conversations. The evaluator compares their answers, mistakes, questions, and turn lengths with patterns in real student conversations. | University | Jun 2026 | Not in cards | |
| EvoArena Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes. | Expert | Jun 2026 | 53.3% | Not in cards |
| EvoArena SWE-Chain-Evo Agents solve software tasks that change across versions. The score is the published accuracy across versioned tasks. | Expert | Jun 2026 | Not in cards | |
| EvoArena Terminal-Bench-Evo Agents solve terminal tasks that change across versions. The score is the published accuracy across versioned tasks. | Expert | Jun 2026 | 65.1% | Not in cards |
| EvoCode-Bench v2 Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit. | Expert | Jun 2026 | 59.1% | Not in cards |
| FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. | University | Jun 2026 | 77.8% | In use |
| FrontierCode Real-repo coding tasks graded for mergeability: correctness, regressions, lint, tests, scope and code quality. Failing any blocker scores zero. | Expert | Jun 2026 | 34.3% | In use |
| FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. | Research frontier | Jun 2026 | 38.1% | In use |
| Game Arena Checkers Language models play American checkers against each other, with forced jumps and crowned kings. Ratings come from head-to-head games. | Everyday | Jun 2026 | | Not in cards |
| Game Arena Clobber Language models play Clobber, where every move captures an adjacent enemy stone. The player who makes the last legal move wins. | Everyday | Jun 2026 | | Not in cards |
| Game Arena Dark Hex Language models play Hex without seeing the opponent's stones, so they must probe cells to learn the hidden board while building a path. | Everyday | Jun 2026 | | Not in cards |
| Game Arena Dots and Boxes Language models take turns drawing lines between dots on a grid. Closing the fourth side of a box claims it. | Everyday | Jun 2026 | 1,393 | Not in cards |
| GameCraft-Bench Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together. | Expert | Jun 2026 | 75.9% | Not in cards |
| GDPval-AA v2 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Outputs are compared pairwise and ranked by Elo. | Expert | Jun 2026 | 1,861 | In use |
| GeneBench Pro Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. | Expert | Jun 2026 | 31.5% | In use |
| Harbor-Index Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. | Research frontier | Jun 2026 | 28.1% | In use |
| HealthAgentBench Agents use terminal tools to complete 54 clinical and biomedical tasks across seven categories. Task-specific verifiers measure success, including a model judge for radiology report correction. | Expert | Jun 2026 | 55% | Not in cards |
| Hedge-Bench Agents analyze supplied company and investment documents for 102 open-ended financial research tasks. A model judge applies rubrics that check whether the reasoning covers the analytical steps identified by investment professionals. | Expert | Jun 2026 | 31.5 | Not in cards |
| IMO-LeanProofBench Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof. | Competition | Jun 2026 | | Not in cards |
| iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. | Everyday | Jun 2026 | 51.9% | Not in cards |
| K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. | Expert | Jun 2026 | 45.7% | Not in cards |
| KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. | Research frontier | Jun 2026 | 76.9% | System cards only |
| Kimi Claw 24/7 Bench Agents work on 17 persistent multi-day professional scenarios through OpenClaw, with 610 checkpoints. The score is the mean pass rate over three runs. | Expert | Jun 2026 | 46.9% | In use |
| Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages. | Expert | Jun 2026 | 77.1% | In use |
| Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. | Expert | Jun 2026 | | In use |
| LLM SoccerArena World Cup 2026 Models forecast World Cup match scores and outcome probabilities before kickoff. The organizers compare forecasts with match results under different prompts and web access rules. | Everyday | Jun 2026 | 168.0 | Not in cards |
| LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. | Expert | Jun 2026 | 0.66 | Not in cards |
| MANDOS Run 003 Models make decisions during seven industrial incidents and four fictional scenarios. A model judge checks their decisions, including whether they seek expert advice. | Expert | Jun 2026 | 100% | Not in cards |
| MarineEVT Models answer 2,000 held-out questions about underwater events, including species interactions, movement, and timing. The authors report accuracy across five reasoning categories. | University | Jun 2026 | | Not in cards |
| MCPMark Verified Agents complete 127 tasks through version-pinned MCP servers. Stabilized verifiers score single-run task success. | University | Jun 2026 | 96.1% | Not in cards |
| MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. | Expert | Jun 2026 | 69.2 | Not in cards |
| MetroLLM-Bench Agents operate a transit kiosk using tools for routes, fares, and service disruptions. The first score checks their actions automatically. The composite also includes model judgments. | Everyday | Jun 2026 | | Not in cards |
| MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. | Research frontier | Jun 2026 | 97.8% | Not in cards |
| MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. | Everyday | Jun 2026 | Not in cards | |
| MyPCBench Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks. | Everyday | Jun 2026 | 62% | Not in cards |
| NatureBench Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results. | Research frontier | Jun 2026 | 23.3% | Not in cards |
| OmniGameArena Models play games while response latency advances the game clock. | Everyday | Jun 2026 | 0.47 | Not in cards |
| Opus Magnum Bench Agents build machines on a hexagonal grid to make a target molecule. Each puzzle score compares the machine’s combined cost, cycles, and area with the human record. | Competition | Jun 2026 | 25.5 | Not in cards |
| OSWorld 2.0 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. | University | Jun 2026 | 20.6% | In use |
| PhoneHarness Bench Agents complete Android workflows through graphical controls, device commands, and structured tools. Checks inspect files, system settings, application state, and unintended side effects. | Everyday | Jun 2026 | | Not in cards |
| PlanBench-XL Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures. | Expert | Jun 2026 | | Not in cards |
| Poker Arena Models play no-limit Texas Hold’em across repeated sessions. The benchmark scores tournament chips and nine strategic dimensions, including memory and bluffing. | Everyday | Jun 2026 | 15730.0 | Not in cards |
| Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. | Expert | Jun 2026 | 94.6% | System cards only |
| PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. | Expert | Jun 2026 | | Not in cards |
| Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. | Expert | Jun 2026 | 76.9% | Not in cards |
| Questflow Traders Arena Agents and people trade on Hyperliquid and Polymarket in a public competition. The leaderboard reports weekly portfolio returns alongside gains and trading volume. | University | Jun 2026 | Not in cards | |
| Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. | Expert | Jun 2026 | 62.1% | Not in cards |
| Ramp SWE-Bench v1 Agents modify Ramp’s backend code to satisfy engineering requests. Hidden tests check whether each patch solves the task. | Expert | Jun 2026 | 88.5% | Not in cards |
| ReactSim-Bench Driving simulators predict how traffic responds when the vehicle changes its behavior. The authors test those reactions in 2,636 scenarios. | University | Jun 2026 | Not in cards | |
| RedlineBench v2 (Fa1zvn) Agents negotiate eight contract terms with a simulated vendor. The grader compares the agreement with the best deal both sides could accept. | University | Jun 2026 | Not in cards | |
| Reward Hack Bench Agents attempt eight coding and puzzle tasks under controlled conditions. A judge distinguishes independent solutions from use of published answers. The control omits hints. | University | Jun 2026 | Not in cards | |
| Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. | Expert | Jun 2026 | 3.90% | System cards only |
| RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. | Everyday | Jun 2026 | | Not in cards |
| Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. | Everyday | Jun 2026 | Not in cards | |
| RTSGameBench Agents play the open real-time strategy game Beyond All Reason. | Expert | Jun 2026 | Not in cards | |
| scBench-Long Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning. | Expert | Jun 2026 | 36.9% | Not in cards |
| scBench-Long (original study) Agents complete 21 tasks in long single-cell data analysis workflows. The authors compare their answers with results from the original analyses. | Expert | Jun 2026 | 25.4% | Not in cards |
| SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. | Expert | Jun 2026 | 58.4% | Not in cards |
| SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. | Everyday | Jun 2026 | | In use |
| Senior Engineer Benchmark Agents rewrite the collaboration system in a frozen copy of the Proof codebase. Every grades their changes against a 100-point rubric. | Expert | Jun 2026 | 91.0 | Not in cards |
| Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate. | Expert | Jun 2026 | 39.3% | Not in cards |
| SentinelBench Agents monitor changing web environments and respond when scripted events make progress possible. Scores measure completion, reaction time and resource use. | Expert | Jun 2026 | Not in cards | |
| Slay the Spire Bench (alavico) Models choose an action from the text state of Slay the Spire. The campaign report measures floors reached and wins at each difficulty. | Everyday | Jun 2026 | 51.0 | Not in cards |
| Snowflake Analytical Search: ArXiv Papers Agents count, group and compare findings across research papers. The evaluation checks numeric and categorical answers against references and scores lists with F1. | University | Jun 2026 | Not in cards | |
| SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. | Everyday | Jun 2026 | Not in cards | |
| SoCRATES Models mediate disputes in 600 scenarios across eight domains. A model judge rates topic agreement to measure consensus gain, intervention timeliness, and intervention effectiveness. | University | Jun 2026 | Not in cards | |
| SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. | Everyday | Jun 2026 | 14.4% | Not in cards |
| SpatialWorld Digital Agents complete entertainment tasks in digital environments. The score is the published digital task success rate. | Everyday | Jun 2026 | | Not in cards |
| SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. | Everyday | Jun 2026 | 14.4% | Not in cards |
| SpreadsheetBench 2 Agents complete 321 expert-annotated business spreadsheet workflows, such as financial modeling, debugging, and charting, in multi-sheet workbooks. Scoring is task accuracy. | Expert | Jun 2026 | 34.9% | In use |
| STATE-Bench Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. | Expert | Jun 2026 | 58.9% | Not in cards |
| Surface Evolver Bench Agents write Surface Evolver datafiles that model liquid surfaces under physical constraints across 16 tasks. Deterministic checks give partial credit for matching hidden reference quantities. | Expert | Jun 2026 | 95% | Not in cards |
| SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail. | Expert | Jun 2026 | 21.5% | Not in cards |
| SWE-Interact Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session. | Expert | Jun 2026 | 50.7% | Not in cards |
| SWE-Marathon 20 realistic multi-hour software tasks (library reproductions, product clones, ML work). Verifiers combine hidden tests, parity and integrity checks. | Expert | Jun 2026 | 29% | In use |
| SWE-Together Coding agents work through 109 repository tasks rebuilt from real multi-turn sessions between users and coding agents. A simulated user replays the original user's goals and gives feedback when the agent needs it. Scores count tasks solved and the number of corrections the user had to make. | Expert | Jun 2026 | 69.3% | Not in cards |
| Terminal-Bench Challenges Agents build a large project end to end with no time limit, such as a faster Rust compiler or a WebGL renderer in WebAssembly. Each challenge has its own test bed and metric. | Research frontier | Jun 2026 | Not in cards | |
| Toolathlon-Verified 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. | University | Jun 2026 | 78.4% | In use |
| ToolMaze Agents complete tool-use tasks while the runtime introduces explicit or hidden tool failures. A judge checks completion across four task structures and five perturbation settings. Metrics report task success, recovery rate, and recovery cost. | University | Jun 2026 | | Not in cards |
| Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. | Expert | Jun 2026 | System cards only | |
| Travel Agent Compassion Agents book travel activities when the customer does not mention animal welfare. The benchmark measures how often they choose a welfare-neutral option. | Everyday | Jun 2026 | 100% | Not in cards |
| TUA-Bench Agents use a terminal for general computer tasks beyond coding, such as file, system and everyday productivity work. | Everyday | Jun 2026 | 65.8% | Not in cards |
| Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. | University | Jun 2026 | 60.2% | Not in cards |
| TxBench-Preclinical-Pharmacology Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision. | Expert | Jun 2026 | 68% | Not in cards |
| UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. | Everyday | Jun 2026 | 55.8% | Not in cards |
| VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output. | Expert | Jun 2026 | | In use |
| Vision2Web S3-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Jun 2026 | 74.6% | Not in cards |
| WeaveBench Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts. | Expert | Jun 2026 | 41.2% | Not in cards |
| WebExploitBench Agents find and exploit 110 vulnerabilities in 15 real web applications, each in its own container. A verifier checks that the agent's proof of concept hits the same endpoint as the reference exploit. | Expert | Jun 2026 | 36.4% | In use |
| τ-Rec Agents interview simulated moviegoers and search a catalog. A recommendation succeeds when it meets the user’s constraints and follows the service policy. | Everyday | Jun 2026 | | Not in cards |
| 3DCodeBench Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. | May 2026 | | In use | |
| AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. | Expert | May 2026 | 0.80 | System cards only |
| Acme Agent Evals Agents use GitHub tools to answer questions about issues, pull requests, and code in a simulated software project. | University | May 2026 | 85.1% | Not in cards |
| AEC-Bench (Theodore Galanos) Agents perform engineering calculations such as cable sizing and hydraulic design. Task verifiers score their outputs, and the leaderboard reports mean reward and completion. | University | May 2026 | 0.89 | Not in cards |
| AerialSim Agents fly a simulated drone using camera observations or write a program for the flight. The simulator checks the recorded trajectory against the mission requirements. | Expert | May 2026 | 73.2% | Not in cards |
| Agent Island Models play multi-agent games that require cooperation, conflict, and persuasion. | Everyday | May 2026 | 0.88 | Not in cards |
| AgentEscapeBench Agents solve procedural escape rooms using unfamiliar tools. The evaluator checks whether they complete dependent steps and escape. | Everyday | May 2026 | 76.9% | Not in cards |
| Agentick Agents solve procedurally generated gridworld tasks involving navigation, planning, reasoning, memory, generalization, and coordination. Scores normalize task performance between random and oracle reference policies. | Everyday | May 2026 | 0.63 | Not in cards |
| AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. | Expert | May 2026 | 38.4% | In use |
| AgentKernelArena Coding agents optimize GPU kernels through full workflows with compilers and profilers. Scores check speedups and whether gains generalize beyond tuned shapes. | Expert | May 2026 | 3.23× | Not in cards |
| AgentThreatBench Agents carry out routine tasks in the presence of untrusted content. The benchmark scores task completion and security separately. | University | May 2026 | Not in cards | |
| AI-Spire (AGI-Eval) Models replay one Slay the Spire 2 seed, build decks, and choose combat moves. The organizers record the floor reached on each attempt. | Everyday | May 2026 | 21.6 | Not in cards |
| Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. | May 2026 | 1.13 | System cards only | |
| Artificial Analysis Coding Agent Index v1.0 Artificial Analysis's first index of coding agents, each a model in a harness, averaged over three equally weighted coding benchmarks. | May 2026 | Not in cards | ||
| ASPI Agents complete simulated tasks with incomplete instructions. The authors compare adherence to user intent across matched versions of each task. | Everyday | May 2026 | | Not in cards |
| AsyncTool Agents coordinate several tool-use tasks while tool responses arrive after simulated delays. The evaluator checks function calls, parameters, task completion, and the final environment state. Scheduling metrics measure whether agents use waiting time effectively. | Everyday | May 2026 | 38.1% | Not in cards |
| Augment Prism multi-turn coding benchmark Coding agents follow a queued conversation and edit a repository. A model compares the final changes with the original pull request. | University | May 2026 | 0.21 | Not in cards |
| AvalancheBench (May study) Agents recover underlying patterns from synthetic enterprise data and report their findings. | Expert | May 2026 | Not in cards | |
| BiomniBench Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer. | Expert | May 2026 | 63.9 | Not in cards |
| Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. | Expert | May 2026 | System cards only | |
| BoardGameBench (Homer Quan) Models play seven board game variants against a search engine. The scorer rewards quick wins and gives partial credit for longer losses. | Everyday | May 2026 | | Not in cards |
| CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. | University | May 2026 | 45.1% | System cards only |
| Cattle Trade Agents play a long economic game with auctions, hidden offers, bargaining, bluffing, and resource limits. | Everyday | May 2026 | Not in cards | |
| CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. | Expert | May 2026 | 54.7% | Not in cards |
| ClawBench V2 Browser agents do 130 new everyday tasks on live websites under six supported harnesses. Request interception and a more lenient LLM judge score each run. | Everyday | May 2026 | 44.6% | Not in cards |
| Cohere North internal evals Cohere internal evals of its North agent platform: agentic question answering, spreadsheet data analysis and memory use. LLM-as-judge scoring. | University | May 2026 | Not in cards | |
| ContractBench (Observation Contracts) Agents complete 33 API workflow tasks. Validators check whether agents preserve returned values exactly and use them while they remain valid. | University | May 2026 | 77.8% | Not in cards |
| CTF-Archive Diamond Agents solve 285 difficult capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. | Expert | May 2026 | 71% | Not in cards |
| CursorBench 3.1 Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.1 added codebase understanding, bug finding, planning and code review problems. Agentic graders score correctness. | Expert | May 2026 | 72.9% | Fading |
| CutVerse Agents complete editing tasks in seven applications, including Premiere Pro, Photoshop, and DaVinci Resolve. Vision models grade progress against milestones in the recorded workflow. | University | May 2026 | 68.3% | Not in cards |
| DataPrep-Bench Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score. | Expert | May 2026 | | Not in cards |
| DataPrep-Bench Finance Models prepare training data for the finance domain. The score is the published downstream average after backbone training. | Expert | May 2026 | 59.3 | Not in cards |
| DataPrep-Bench General Models prepare training data for the general domain. The score is the published downstream average after backbone training. | Expert | May 2026 | 78.2 | Not in cards |
| DataPrep-Bench Law Models prepare training data for the law domain. The score is the published downstream average after backbone training. | Expert | May 2026 | 76.4 | Not in cards |
| DataPrep-Bench Math Models prepare training data for the math domain. The score is the published downstream average after backbone training. | Expert | May 2026 | 25.5 | Not in cards |
| DataPrep-Bench Medical Models prepare training data for the medical domain. The score is the published downstream average after backbone training. | Expert | May 2026 | 40.5 | Not in cards |
| DataPrep-Bench Science Models prepare training data for the science domain. The score is the published downstream average after backbone training. | Expert | May 2026 | | Not in cards |
| DeepSWE 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. | Expert | May 2026 | 74.1% | In use |
| DeepTerminalBench Agents get one attempt at each of 50 deep engineering tasks in Docker, in the Terminal-Bench 2.0 format. A task passes only if every hidden test passes. Tasks were calibrated so Claude Opus 4.6 still fails some at pass@4. | Expert | May 2026 | 34% | Not in cards |
| DTap Direct Misuse An attacker agent generates adaptive jailbreaks against a target agent across tool-use and browser-use tasks. Reports attack success rate. | Expert | May 2026 | 21.2% | System cards only |
| DTap Prompt Injection An attacker agent plants indirect prompt injections in tool and browser environments against a benign-user agent. Reports attack success rate. | Expert | May 2026 | 8.60% | System cards only |
| ERP-Bench Agents complete 300 procurement and manufacturing workflows in Odoo 19. A constraint solver generates the tasks and optimal solutions, and the verifier checks final business state. | Expert | May 2026 | 43.4% | Not in cards |
| ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. | Everyday | May 2026 | 70% | Not in cards |
| euchre-bench Coding agents build a euchre bot and test it against four opponents. Each opponent contributes up to 100 wins to the total. | University | May 2026 | 140.0 | Not in cards |
| EVA-Bench Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes. | Everyday | May 2026 | 0.74 | In use |
| EvoCode-Bench Coding agents keep one project working across 26 tasks of 5 to 15 rounds each, while requirements change, pile up and conflict. Cumulative hidden tests check every round in a persistent container. | Expert | May 2026 | 54% | Not in cards |
| ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. | Expert | May 2026 | 100% | In use |
| ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. | Expert | May 2026 | 42.4% | In use |
| Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. | Expert | May 2026 | | System cards only |
| Game Arena Word Association Teams of language models give one-word clues and guess linked words, in a game like Codenames, against an opposing team. | Everyday | May 2026 | | Not in cards |
| GBAEval Agents get 24 hours to write a Game Boy Advance emulator in Rust for WebAssembly. Replay, hardware, and audio tests compare it with Mesen2. | Research frontier | May 2026 | 90.6% | Not in cards |
| Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. | May 2026 | | System cards only | |
| harness-bench-fast Agents operate on files and code using tools and memory. Harbor exports 231 tasks from version 0.3.0. Later public leaderboards report Light and Medium suites separately. | University | May 2026 | Not in cards | |
| HarnessAudit-Bench Agents run inside execution harnesses such as Claude Code, Codex and OpenClaw on 210 tasks across eight domains, from finance to software engineering, with safety rules built in. The audit reads the full trajectory through channels the agent cannot change. It scores whether the agent stays inside its permission and information-flow policies, completes checkpoints, and keeps both under prompt injection, vague goals and tool errors. | Expert | May 2026 | | Not in cards |
| Harvey LAB (Legal Agent Benchmark) Agent works through legal matter files across 24+ practice areas and produces work product. Expert rubric, task passes only if every criterion passes. | Expert | May 2026 | | In use |
| HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. | Everyday | May 2026 | Not in cards | |
| HWE Bench V1 (RISC-V CPU design) Agents improve a RISC-V processor in Verilog. Designs must pass correctness checks before the benchmark ranks them by estimated CoreMark iterations per second. | Expert | May 2026 | 983.2 | Not in cards |
| HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials. | Expert | May 2026 | | System cards only |
| ITBench-AA Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON. | Expert | May 2026 | Not in cards | |
| JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. | Expert | May 2026 | 61.4% | In use |
| Lazy investigation on misleading codebases (Anthropic) Models trace small, intentionally misleading codebases to answer how a value is set. Score is the share of lazy investigations ending in a wrong answer. | Expert | May 2026 | 0% | System cards only |
| Local LLM Tetris Test Agents build Tetris in one HTML file and can revise it after playing. A separate Claude agent checks piece movement, line clearing, scoring, and restart. | University | May 2026 | | Not in cards |
| LongMemEval-V2 Memory systems help agents learn the affordances, dynamics and failure modes of customized web environments. Questions check whether that experience was internalized. | Expert | May 2026 | | Not in cards |
| Maze Bench Agents explore connected rooms in a 3D puzzle game. Each of the 100 gems contributes one percentage point to the score. | Expert | May 2026 | 24% | Not in cards |
| MBABench Agents build complete financial models in Excel from a case prompt and documents. An LLM judge grades accuracy, formulas and formatting on a 0 to 100 scale. | Expert | May 2026 | 31.2 | Not in cards |
| Megaminx solver Models solve a simulated twelve-faced twisty puzzle. They read the puzzle state as text, make moves with tools and receive rewards from the simulator. | University | May 2026 | Not in cards | |
| MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. | Competition | May 2026 | | System cards only |
| MLS-Bench Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. | Research frontier | May 2026 | 50.3% | In use |
| MobileGym Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state. | Everyday | May 2026 | | Not in cards |
| MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. | Expert | May 2026 | 16.2% | Not in cards |
| NanoGPT-Bench Coding agents with a fixed H100-hour budget and no internet try to improve on a strong NanoGPT speedrun starting point. The score is the share of later human progress recovered. | Research frontier | May 2026 | 9.30% | Not in cards |
| NARRA-Gym Agents build interactive stories around a user’s experiences and adapt scenes as the conversation unfolds. Judges rate story quality and user experience separately. | Everyday | May 2026 | 2 | Not in cards |
| OpenComputer Computer-use agents complete 1,000 desktop tasks across 33 real applications, including browsers, office tools, creative software, development environments and messaging apps. App-specific verifiers inspect the final application state, and the harness gives partial credit from the recorded trajectory. | Everyday | May 2026 | 68.3% | Not in cards |
| Parametric CAD Bench V1 Agents write FreeCAD scripts that build editable parametric parts from plain-English descriptions and dimension tables. Each of 100 tasks scores the harmonic mean of geometry similarity and spec consistency. | Expert | May 2026 | In use | |
| PDEAgent-Bench Models write finite element solvers for partial differential equations. A solver must run successfully, meet a numerical error tolerance, and finish within its time limit. | Expert | May 2026 | | Not in cards |
| PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. | Expert | May 2026 | 50% | System cards only |
| PinchBench 2.0 Models drive an OpenClaw agent through 148 real-world tasks, including data and log analysis, meeting notes, DevOps, PDFs, research and writing. Graded by checks and a Haiku judge. | Everyday | May 2026 | 93.5% | Not in cards |
| ProgramBench 200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original. | Research frontier | May 2026 | 18.5% | In use |
| Prompt injection live bug bounty (Anthropic with Gray Swan) Expert red teamers submit prompt injection attacks against hidden frontier models in tool use, coding and computer use scenarios. Score is attack success rate. | Expert | May 2026 | 0.08% | System cards only |
| PTCG-Bench Agents play the Pokémon Trading Card Game by choosing legal actions from a partially visible board. Match results determine their Glicko-2 ratings. | School | May 2026 | | Not in cards |
| QF-Bench Agents write and run code for 87 quantitative finance tasks in a network-isolated sandbox with real data. Tasks cover derivatives pricing, fixed income, credit, risk, factor research, backtesting and market microstructure. More than 3,300 assertions check numeric outputs against finance rules such as no-arbitrage bounds and put-call parity. Scored by pass@1 over three runs. | Expert | May 2026 | 61.7% | Not in cards |
| ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. | Research frontier | May 2026 | | Not in cards |
| RoadmapBench Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests. | Expert | May 2026 | 39.1% | In use |
| RoboChallenge Table30 v2 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. | Everyday | May 2026 | 0.73% | Not in cards |
| RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. | Everyday | May 2026 | 21.5% | Not in cards |
| Rule2DRC Models write KLayout scripts from chip design rules. The evaluator checks whether each script flags the correct violations on every test layout. | Expert | May 2026 | 67.9% | Not in cards |
| SaaS-Bench 106 long-horizon GUI workflows across 23 locally deployed real SaaS apps. Scored from app databases with weighted checkpoints and full-resolution rate. | University | May 2026 | 31.1% | In use |
| ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change. | Expert | May 2026 | Not in cards | |
| Science Sim Chemistry Models compare simulated compounds and choose whether to advance or reject them. The evaluator scores decisions against measured reaction outcomes and a budget. | University | May 2026 | Not in cards | |
| SEC-bench Pro Agents hunt and reproduce security bugs in large software projects from limited information. The first release covers V8 and SpiderMonkey. | Expert | May 2026 | | In use |
| SIGHT (Search and Intent Gauge for Hospitality and Tourism) Agents recommend restaurants, hotels and attractions from user preferences and search results. Evaluators check constraints, evidence and privacy. | Everyday | May 2026 | Not in cards | |
| Skill Reward Hacking Researchers track when models learn to increase a proxy reward without improving task quality. They measure warning time, false alarms, and transfer across reward functions. | University | May 2026 | | Not in cards |
| Slitherlink (savi) Models place edges to form a single loop on seven-by-seven Slitherlink boards. The evaluation uses ten boards disjoint from its 15 training examples. | School | May 2026 | Not in cards | |
| SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. | Expert | May 2026 | 40.2% | Not in cards |
| SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. | Expert | May 2026 | 40.5% | Not in cards |
| SpatialBench-Long (original study) Agents complete 24 tasks in long spatial transcriptomics workflows. The authors compare their answers with results from the original analyses. | Expert | May 2026 | 11.1% | Not in cards |
| STATE-Bench Main 0.7.0 Agents complete enterprise tasks in travel, customer support, and shopping. The score is overall pass@1 for this track and version. | Expert | May 2026 | 55.7% | Not in cards |
| Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses. | Expert | May 2026 | | Fading |
| SWE Atlas Refactoring Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result. | Expert | May 2026 | 59% | In use |
| TeachArena Agents make teaching decisions, tutor students through dialogue, and carry out learning management system tasks. The evaluator checks conversation turns, artifacts, and final system states. | University | May 2026 | 0.80 | Not in cards |
| Terminal-Bench 2.1 89 hard terminal tasks across 16 categories in containers. Hidden tests verify the final state. 5 trials per task. | Expert | May 2026 | 87.4% | In use |
| Terminal-X A suite of three terminal datasets for coding agents. DeepTerminalBench tests single-shot depth, EvoCode-Bench tests multi-round iteration and RoadmapBench tests version upgrades on real repositories. | Expert | May 2026 | 42.5% | Not in cards |
| TerminalWorld Agents complete terminal workflows reconstructed from public recordings in isolated environments. Automated tests check the resulting files and system state. A manually reviewed 200-task subset supports the published model comparisons. | University | May 2026 | 62.5% | Not in cards |
| TERMS-Bench Agents negotiate the price of a single item with simulated counterparts over up to 10 rounds of alternating offers. Counterparts follow six behavior families across three regimes. A program scores each agent from its logged actions on four separate axes: surplus captured, agreement calibration, belief error about the opponent and procedural violations. Extra modes score merchant profit and a bankroll over many sessions. | Everyday | May 2026 | 0.69 | Not in cards |
| TVRBench Agents move through simulated indoor scenes to reproduce a target image's viewpoint. Success requires stopping at exactly the target pose on the simulator's discrete grid. The evaluation split has 500 tasks across 80 scenes. | Everyday | May 2026 | | Not in cards |
| UniGUI-Bench Agents interpret and operate graphical interfaces. The benchmark tests 11 capabilities and includes separate system evaluations. | Everyday | May 2026 | 80.9% | Not in cards |
| Vals Finance Agent v2 927 expert-reviewed research questions on public-company filings. Partial credit only after required checks pass. Private test split. | Expert | May 2026 | | In use |
| Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. | Expert | May 2026 | 74.2% | Not in cards |
| Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Expert | May 2026 | 70.7% | Not in cards |
| ViBench 2.0 Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans. | Expert | May 2026 | 64.7% | Not in cards |
| VitaBench 2.0 Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics. | Everyday | May 2026 | 50.3% | In use |
| WebHarbor Agents navigate local copies of 153 websites to search, sign in, book services, and change stored state. The suite contains 3,518 tasks, including 306 verified test tasks and 2,288 training tasks. The test score measures the percentage of tasks passed by scripted verifiers. | Everyday | May 2026 | 86.7% | Not in cards |
| Workspace-Bench-Lite Agents complete 100 workspace tasks. The leaderboard reports the percentage of 1,856 rubric checks passed, with separate runs for each model and harness. | University | May 2026 | 66.7% | Not in cards |
| WorkSpaceBench Agent completes 388 tasks in realistic worker file workspaces (20k files) needing cross-file dependency reasoning. Scored against 7,399 rubrics. | University | May 2026 | | In use |
| WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. | May 2026 | | Not in cards | |
| Zerolang Editing Agents repair and refactor Zerolang programs through graph editing tools. The evaluation includes 67 held-out synthetic tasks. | University | May 2026 | Not in cards | |
| τc-Bench Agents use tools while conversing with simulated customers in airline, retail, and telecom tasks. The evaluator compares the final environment state with the expected state. Generated scenarios add unusual tool combinations, ambiguous requests, and distracting database records. | Everyday | May 2026 | | Not in cards |
| AARRI-Bench Agents complete research intern tasks that test scientific judgment, professional conduct, and attention to detail. The Harbor collection contains 82 tasks. | Expert | Apr 2026 | 68.3% | Not in cards |
| Agent Bazaar Agents price goods or trade with sellers in simulated markets. The benchmark measures whether their decisions sustain prices, deter fraud, and preserve welfare and profits. | University | Apr 2026 | Not in cards | |
| AlphaEval Agent products such as Claude Code and Codex do 94 production tasks from 7 companies across 6 O*NET domains. Grading mixes formal checks, UI tests and rubrics. | Expert | Apr 2026 | 64.4% | Not in cards |
| AlphaZero Connect Four study Coding agents implement an AlphaZero pipeline for Connect Four. The resulting programs play a round-robin tournament against the Pascal Pons solver. | Expert | Apr 2026 | 1,938 | Not in cards |
| AutomationBench Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. | University | Apr 2026 | | In use |
| AutoResearchBench Agents search the scientific literature, either tracking down one target paper through multi-step probing or collecting every paper that meets given conditions. | Expert | Apr 2026 | 9.39% | Not in cards |
| BankerToolBench 100 end-to-end junior-banker tasks producing Excel, PowerPoint and Word deliverables. Agentic grader scores ~150 weighted expert rubric criteria. | Expert | Apr 2026 | 16% | In use |
| BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. | Expert | Apr 2026 | 83.6% | System cards only |
| BRIGHT-Pro Models retrieve passages that cover different parts of a question. The 739 questions include 115 from Earth Science Stack Exchange. | University | Apr 2026 | | Not in cards |
| BTF-2 Agents forecast 1,417 binary questions asked in October 2025 and resolved in December 2025, researching offline in a frozen corpus of 15 million documents. Scored by Brier score. | Expert | Apr 2026 | 0.13 | Not in cards |
| Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. | Expert | Apr 2026 | 97.1% | Fading |
| Clash of AIs Season 1 Models publish trade entries, targets, and stops before prices move. Clash of AIs simulates fills against market prices and ranks accounts by total equity. | University | Apr 2026 | $283,903 | Not in cards |
| Claw-Eval Pass^3 Agents complete 300 human-verified tasks in service orchestration, multimodal interaction, and professional dialogue. Pass^3 credits a task only if all three trials pass. | University | Apr 2026 | | In use |
| Claw-Eval-Live Agents complete 105 workflow tasks in 17 families drawn from live ClawHub workflow signals, scored on both the workflow and the final workspace state. | University | Apr 2026 | 66.7% | Not in cards |
| Claw-Eval-MM Agents complete the 101 multimodal tasks of Claw-Eval, such as video questions, document extraction, and webpage generation. Rubric items graded from run traces set scores. | University | Apr 2026 | 54.7% | In use |
| ClawBench Browser agents do 153 everyday tasks on live websites, such as booking travel, ordering food and applying for jobs. Request interception stops real side effects, and an LLM judge scores the five-layer traces. | Everyday | Apr 2026 | 61.4% | Not in cards |
| CocoaBench Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output. | Expert | Apr 2026 | 45.1% | Not in cards |
| CodeIPI Coding agents fix bugs while handling untrusted repository text. The benchmark scores task completion, resistance to embedded instructions and false alarms on clean tasks. | University | Apr 2026 | 91.1% | Not in cards |
| Cooperate to Compete (C2C) Multi-player conquest game with private non-binding negotiation and secret asymmetric objectives. LM agents and humans are ranked by Plackett-Luce player strength from their game results. | Everyday | Apr 2026 | | Not in cards |
| CUA-World-Long Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task. | Expert | Apr 2026 | 27.5% | Not in cards |
| DataSciBench Agents complete open-ended data-science tasks that require data cleaning, analysis, visualization and modeling. Component checks and overall task success score the work. | Expert | Apr 2026 | Not in cards | |
| DeepSeek Chinese white-collar tasks evaluation Agents complete 30 Chinese professional analysis, document generation and editing tasks across 13 industries. Blind human annotators compare outputs against a baseline model. | Expert | Apr 2026 | | System cards only |
| DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task. | Expert | Apr 2026 | | System cards only |
| DeepSeek search Q&A evaluation Models answer objective and subjective Chinese search questions with retrieval or agentic search. Pairwise judging gives win, loss and tie rates. | Everyday | Apr 2026 | | System cards only |
| DELEGATE-52 Models perform long delegated document-editing workflows across 52 professional domains. Scores measure artifact fidelity after repeated transformations. | Expert | Apr 2026 | Not in cards | |
| Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. | Expert | Apr 2026 | 83% | Fading |
| FrogsGame post-training task Agent has 20 hours to post-train Qwen3-8B to solve FrogsGame puzzle boards. A hidden verifier scores the best checkpoint on unseen boards. | Research frontier | Apr 2026 | 34% | Not in cards |
| FrontierSWE v1 Agents work up to 20 hours on 17 ultra-long engineering, performance and research tasks in their own harnesses. The board ranks models by average task rank. The score here is dominance, the win rate against a random baseline, which labs report. | Research frontier | Apr 2026 | 88.2% | In use |
| Full-Duplex-Bench v3 Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking. | Everyday | Apr 2026 | | Fading |
| Gaia2-CLI Agents complete realistic tasks through command-line tools. The benchmark measures task success across search, execution, adaptability, ambiguity, and time constraints. | Everyday | Apr 2026 | 57% | Not in cards |
| Game Arena Five in a Row Language models play a gomoku-style game and try to line up five of their tokens before the opponent does. | Everyday | Apr 2026 | | Not in cards |
| GameWorld Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. | Everyday | Apr 2026 | 76.8% | System cards only |
| GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. | Expert | Apr 2026 | 33.2% | Fading |
| GeoNatureAgent Benchmark Agents use a geospatial API to analyze environmental indicators in Spain and Portugal. The grader checks tool use and answers against task requirements. | University | Apr 2026 | 60.8% | Not in cards |
| GomokuBench Models place stones on a 19 by 19 board against a search engine. The score rewards wins, quicker wins, and longer losses. | Everyday | Apr 2026 | | Not in cards |
| GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. | Expert | Apr 2026 | 11.4% | Not in cards |
| Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. | Research frontier | Apr 2026 | 7.60% | System cards only |
| Hiding a secret password (Anthropic) Models keep a given password secret while an investigator model tries to extract it over many turns. Measures survival and per-turn leak rates. | Expert | Apr 2026 | System cards only | |
| HiL-Bench Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary. | Expert | Apr 2026 | | Not in cards |
| HWE-Bench v1.0 (RTL bug repair) Agents fix bugs in six open-source hardware projects. The benchmark reports the share of fixes that pass repository tests in simulation. | Expert | Apr 2026 | 79.9% | Not in cards |
| Hy-Backend Coding agents complete Tencent in-house backend development tasks. Task and scoring details are not published. | Expert | Apr 2026 | | Fading |
| Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task. | Expert | Apr 2026 | | In use |
| Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published. | Expert | Apr 2026 | | Fading |
| ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. | Expert | Apr 2026 | 30.7% | Fading |
| Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. | Research frontier | Apr 2026 | 81.8% | System cards only |
| KellyBench Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth. | Expert | Apr 2026 | -7.90% | Not in cards |
| Keystone-Eval Agents configure Dockerfiles and test entrypoints for arbitrary code repositories, then run the repositories' tests. | Expert | Apr 2026 | Not in cards | |
| KnowU-Bench Agents use Android apps such as Mattermost and Taodian. They infer preferences, ask for missing details, and decide when to act or stay silent. | Everyday | Apr 2026 | 60.4% | Not in cards |
| LinuxArena AI-control setting in live Linux production-like environments. Measures stealthy side-task completion against action and CoT monitors. | Expert | Apr 2026 | 0% | System cards only |
| Malicious agentic influence campaigns (Anthropic) Agents receive requests to run multi-step influence operations with tools. Score is the refusal rate. | Everyday | Apr 2026 | System cards only | |
| Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. | Expert | Apr 2026 | | System cards only |
| MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE. | Expert | Apr 2026 | 77.1% | Fading |
| Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. | Expert | Apr 2026 | 2.60% | System cards only |
| MMSearch Multimodal agents search and answer questions that require visual and textual evidence. | Expert | Apr 2026 | 72.9% | Fading |
| neuralCAD-Edit Agents edit existing CAD models from text or recorded instructions. CAD experts rate whether they would accept each edit. Software also compares the resulting geometry. | University | Apr 2026 | 0.25 | Not in cards |
| o11y-bench Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment. | Expert | Apr 2026 | 93.8% | Not in cards |
| OccuBench Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable. | Expert | Apr 2026 | 79.6% | Not in cards |
| OpenGameEval Expanded Eval Set Agents extend existing Roblox games and build features from empty scenes. Roblox added 40 coding tasks to the original 47. | University | Apr 2026 | 50.3% | Not in cards |
| Pare-Bench Proactive assistants infer user goals and act across stateful digital applications with simulated users and multi-app task environments. | Expert | Apr 2026 | Not in cards | |
| PHMForge Agents analyze battery and aircraft engine data to diagnose faults, estimate remaining life, and make maintenance decisions. Harbor packages 99 industrial scenarios as 123 tasks. | Expert | Apr 2026 | 80.8% | Not in cards |
| QuantSightBench Models give 90% prediction intervals for 1,000 numerical forecasting questions with known outcomes. The main setting allows retrieval from a fixed news corpus. Mean Log Interval Score penalizes wide intervals and missed outcomes, while coverage measures how often intervals contain the answer. | Expert | Apr 2026 | 6.70 | Not in cards |
| QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. | Everyday | Apr 2026 | 60.5% | Fading |
| RECON Research agents find and verify several facts about identified people on the open web without mixing up people with similar names. | Expert | Apr 2026 | 36.2% | Not in cards |
| Riemann-bench 25 private research-mathematics problems from professors and IMO medalists. Agents with tools, 100 runs per problem. | Research frontier | Apr 2026 | 83.2% | In use |
| RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. | Everyday | Apr 2026 | 46.2% | In use |
| Spatial-Gym Models find paths that satisfy grid puzzle rules. The evaluation uses 500 SPaRC puzzles in one-shot and stepwise settings, including a setting that allows backtracking. | School | Apr 2026 | Not in cards | |
| SpireBench (HermesBridge trial-v0) Models play Slay the Spire 2 from structured game states. Each plays all five characters at the base difficulty, with one run per character. | Everyday | Apr 2026 | 28.0 | Not in cards |
| Supabase Evals Agents build, deploy, investigate, and repair Supabase projects. The hub port uses OpenCode and runs the original task scorers from a pinned release. | Expert | Apr 2026 | 92.8% | Not in cards |
| SWE Atlas Test Writing Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests. | Expert | Apr 2026 | 67% | In use |
| SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. | Expert | Apr 2026 | | System cards only |
| TarantuBench Agents complete simulated web security challenges. Each challenge has a binary success criterion. | Expert | Apr 2026 | Not in cards | |
| The Amazing Agent Race Agents navigate websites, call tools, and calculate a passcode across 1,400 scavenger hunts. The benchmark includes 800 sequential puzzles and 600 with branching paths. | Expert | Apr 2026 | 36.6% | Not in cards |
| Toloka Arena Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed. | Expert | Apr 2026 | 33% | Not in cards |
| UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. | Research frontier | Apr 2026 | 80% | System cards only |
| UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. | Expert | Apr 2026 | 95% | System cards only |
| UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. | Expert | Apr 2026 | 0% | System cards only |
| UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. | Expert | Apr 2026 | 0% | System cards only |
| VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. | Everyday | Apr 2026 | | Not in cards |
| Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. | Expert | Apr 2026 | System cards only | |
| VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. | Expert | Apr 2026 | Not in cards | |
| VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. | Research frontier | Apr 2026 | System cards only | |
| AEC-Bench (Nomic) Agents search construction drawings and specifications to answer questions and identify conflicts. Task verifiers compare their outputs with known findings and award partial credit. | University | Mar 2026 | 70.2 | Not in cards |
| AndroidBench Android development tasks from popular open-source apps: agent resolves issues or builds features, verified by test suites in emulator environments. | Expert | Mar 2026 | 91.8% | In use |
| ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. | Everyday | Mar 2026 | 100% | In use |
| AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. | Expert | Mar 2026 | 76.5% | Not in cards |
| ATM-Bench Agents answer questions about a person’s history using images, videos, and email records. The Harbor package contains 31 hard questions with schema-guided memory. | University | Mar 2026 | 86% | Not in cards |
| AutoResearch An agent edits a small nanochat training script and runs 5 minute training experiments overnight. Scored by final validation bits per byte, where lower is better. | Research frontier | Mar 2026 | Not in cards | |
| BarterBench Agents trade scarce goods to meet inventory targets. The hub port assigns one model-controlled trader to a marketplace of scripted opponents. | University | Mar 2026 | Not in cards | |
| BIBIMBAP Agents read and edit IFC building files. Executable tests compare their changes and answers with reference results. | University | Mar 2026 | 50.2% | Not in cards |
| BigLaw Bench: Research Agents use search tools to find US case law and give cited answers. The tasks span corporate law, litigation and other practice areas. | Expert | Mar 2026 | Not in cards | |
| BixBench-Hypothesis Agents analyze existing datasets to test 51 hypotheses. A rubric awards points for intermediate results and the final conclusion. | Expert | Mar 2026 | Not in cards | |
| BrokenArXiv Models are asked to prove statements from recent arXiv papers that were changed to be plausible but false. Full credit goes to models that say the statement is false. | Research frontier | Mar 2026 | | In use |
| CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. | University | Mar 2026 | | Not in cards |
| Claw-Eval 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. | Everyday | Mar 2026 | 81.4% | In use |
| Clocktower Radio Models play the social deduction game Blood on the Clocktower (the Trouble Brewing script) in mirrored pairs of games. In each match one model plays good and the other evil, and then they swap. A coded Storyteller runs the game, and a Bradley–Terry fit over all matches gives ratings where 1500 is average. | Everyday | Mar 2026 | 1,708 | Not in cards |
| Cooperative Codenames (Max Bittker) Two models cooperate as cluegiver and guesser to identify target words on a hidden board. The default evaluation uses 200 separately generated boards. | Everyday | Mar 2026 | Not in cards | |
| Crate Bench Models identify an album and its artist from the cover image. The benchmark provides 300 test covers and supports closed-book and web-search modes. | Everyday | Mar 2026 | Not in cards | |
| CTI-REALM Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry. | Expert | Mar 2026 | 63.7% | Not in cards |
| CursorBench Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.0 covers edit, refactor and bugfix problems. Agentic graders score correctness, with cost reported. | Expert | Mar 2026 | 61.3% | System cards only |
| Data Agent Benchmark Agents answer business questions over data split across several database systems, with duplicated and inconsistent records and key facts buried in text. The 104 queries cover 17 datasets and four database systems, based on a study of companies building data agents in six industries. Scores are pass@1 against known answers. | Expert | Mar 2026 | | Not in cards |
| Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. | Expert | Mar 2026 | System cards only | |
| EnterpriseOps-Gym Agents do 1,150 expert-written tasks in eight enterprise domains, such as IT, HR and customer service, using 512 tools over live MCP servers. SQL checks on the final database state score each task. | Expert | Mar 2026 | 45.9% | In use |
| FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. | University | Mar 2026 | 81% | Not in cards |
| FinToolBench Agents answer 295 finance questions by selecting and executing tools from a library of 760 financial interfaces. The evaluation separates tool execution, answer correctness, and mismatches in timeliness, intent, and regulatory domain. Answer checks combine model judging with exact matching. | Expert | Mar 2026 | | Not in cards |
| FLTEval Agents complete the formal Lean proofs and define the new mathematical concepts in real pull requests to the Fermat's Last Theorem (FLT) formalization project. Lean and SafeVerify check each submitted diff. | Research frontier | Mar 2026 | | In use |
| Gray Swan IPI Arena benchmark Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. | Mar 2026 | | In use | |
| GTO Wizard Benchmark Models play heads-up no-limit Texas Hold’em against GTO Wizard AI. AIVAT reduces variance while the benchmark reports poker winnings. | Expert | Mar 2026 | -9.23 | Not in cards |
| GUI-CEval Models answer questions about Chinese app screens, locate controls, and choose actions. Agents also complete tasks on physical devices, where the grader checks interface elements. | Everyday | Mar 2026 | | Not in cards |
| InteractiveBench Texas Hold’em Models play no-limit Texas Hold’em from partial observations of cards, chip stacks, and recent actions. Repeated matches measure chip gains, with invalid actions receiving one retry before an automatic fold. | Competition | Mar 2026 | | Not in cards |
| InteractiveBench Trust Game Models choose whether to cooperate or defect in repeated prisoner’s dilemma matches. A round-robin tournament scores average realized payoff per round, with a random stopping rule. | Everyday | Mar 2026 | | Not in cards |
| Kaggle Standardized Agent Exams Agents answer 16 questions on multistep reasoning and handling manipulative prompts. Kaggle publishes an overall score. | School | Mar 2026 | Not in cards | |
| KARLBench Agents answer questions using vector search over fixed document collections. The suite measures six kinds of evidence gathering and reasoning. | University | Mar 2026 | 67.5 | Not in cards |
| LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. | Everyday | Mar 2026 | 71.4% | Not in cards |
| LiveClawBench Agents act as OpenClaw personal assistants on 134 multi-step tasks across 10 app domains backed by stateful mock services. A run passes when its reward exceeds 0.8. | Expert | Mar 2026 | 76% | In use |
| MADQA Agents search 800 PDF documents to answer 2,250 human-written questions. Scoring covers answer accuracy and how much search effort the agent spends. | Everyday | Mar 2026 | | In use |
| ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. | Everyday | Mar 2026 | Not in cards | |
| MM-ClawBench Agents complete common OpenClaw tasks such as study planning, office documents, scheduled research, and code maintenance. Scoring is accuracy. | Everyday | Mar 2026 | 75.4% | Fading |
| Monthly-SWEBench Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1. | Expert | Mar 2026 | 76% | Not in cards |
| MyClaw Bench Models run inside the OpenClaw agent on 45 tasks in four difficulty tiers built from real usage data. Some tasks use an LLM judge. | Everyday | Mar 2026 | Not in cards | |
| NYT Connections (Prime Community) Models identify four groups of related words in New York Times Connections puzzles. The default 100-puzzle evaluation scores completion, efficiency, partial progress, and format. | Everyday | Mar 2026 | Not in cards | |
| OfficeQA Pro Models answer the 133 hardest OfficeQA questions by finding tables in U.S. Treasury Bulletin documents and computing exact numeric answers. | University | Mar 2026 | 60.9% | In use |
| OpenGameEval Debug Evals Agents diagnose and fix faults in Roblox Studio games. The authors created 30 debugging tasks from 15 scenarios. | University | Mar 2026 | 64.7% | Not in cards |
| PE Civil Bench Models solve 110 multiple-choice and 40 numerical civil engineering exam problems. A model compares each answer with the reference solution. | University | Mar 2026 | 87.3% | Not in cards |
| Pencil Puzzle Bench Models solve 300 pencil puzzles of 20 varieties with unique solutions, either in one shot or agentically with step checks. Every board state can be verified against the rules. | Expert | Mar 2026 | 97.6% | Not in cards |
| PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. | Everyday | Mar 2026 | 75.1% | Not in cards |
| PokeAgent Challenge Agents compete in Pokemon battles under partial observability in a Battling Track and play the Pokemon RPG in a Speedrunning Track that needs long-horizon planning. | Expert | Mar 2026 | Not in cards | |
| PostTrainBench CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Research frontier | Mar 2026 | 34.1% | In use |
| RetailBench Agents operate a simulated supermarket for up to 180 days. They choose suppliers, order stock, and set prices as goods age and customers respond. | University | Mar 2026 | $24,351 | Not in cards |
| RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. | Everyday | Mar 2026 | 10.4% | Not in cards |
| RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. | Everyday | Mar 2026 | | Not in cards |
| RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. | Everyday | Mar 2026 | | Not in cards |
| RVMS-Bench Agents search open-domain videos and localize moments using global impression, key moment, temporal context and auditory memory cues. | Expert | Mar 2026 | Not in cards | |
| SlopCodeBench Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity. | Expert | Mar 2026 | 32.1% | Not in cards |
| SWE Atlas Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. | Expert | Mar 2026 | 63.2% | In use |
| SWE-CI Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes. | Expert | Mar 2026 | 92% | Not in cards |
| tau3-bench Agents use tools and follow policies while talking with simulated users in airline, retail, telecom, and banking domains. Scoring is task success rate. | Everyday | Mar 2026 | 96% | In use |
| VAKRA Agents answer multi-step questions by calling more than 8,000 locally hosted APIs over real databases in 62 domains, along with matching document collections. Tasks need chains of 3 to 7 steps across APIs and documents. A deterministic evaluator replays tools and checks the full trajectory. | Everyday | Mar 2026 | | Not in cards |
| Vibe Code Bench End-to-end web application development from natural-language specs. Built apps are tested against functional requirements. | Expert | Mar 2026 | 93.8% | In use |
| Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Mar 2026 | 82.3% | In use |
| Vision2Web S1-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Mar 2026 | 52.7% | Not in cards |
| Vision2Web S2-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Mar 2026 | 43.5% | Not in cards |
| WildClawBench 60 original in-the-wild tasks run inside a live OpenClaw assistant environment, chaining 10-60+ tool calls. Scored by task-level graders. | University | Mar 2026 | 67.2% | In use |
| ZClawBench Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use. | Expert | Mar 2026 | 57.6% | Fading |
| τ-Voice Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction. | Everyday | Mar 2026 | 81.7% | In use |
| τ³-bench Banking Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. | Everyday | Mar 2026 | | In use |
| AD-Bench Agents answer advertising analytics requests with production-platform tools. A model judge checks final answers against refreshed reference answers and measures coverage of expert tool-call trajectories. Results report success over one or three attempts. | University | Feb 2026 | | Not in cards |
| Agent-Diff Agents use code to complete tasks across replicas of Slack, Linear, Box, and Calendar. The evaluator compares the resulting application state with the expected changes. | University | Feb 2026 | | Not in cards |
| Agentic code behavior scores (Anthropic) Agents work through over one hundred realistic coding scenarios. Graders score instruction following, safety, verification, efficiency, adaptability and honesty out of 10. | Expert | Feb 2026 | System cards only | |
| AI GameStore Vision-language models play short episodes of 100 games adapted from Apple App Store and Steam top charts. Each game score is divided by the average human score, and models are ranked by the geometric mean. | Everyday | Feb 2026 | | Not in cards |
| AIDABench Agents analyze documents and spreadsheets to answer questions, draw charts, and generate files across 603 tasks. They can execute Python in a sandbox. Separate model evaluators check answer correctness, chart correctness and readability, and spreadsheet structure and content. | Expert | Feb 2026 | 59.4% | Not in cards |
| AIRS-Bench Agents carry out 20 AI research tasks taken from recent ML papers, without baseline code. Scores are normalized so that 1.0 matches the published human state of the art. | Research frontier | Feb 2026 | 0.40 | System cards only |
| ArXivMath Final-answer research-level math problems extracted monthly from recent arXiv papers and filtered to be self-contained. | Research frontier | Feb 2026 | | In use |
| Audio Arena Speech-to-speech models hold one long spoken conversation per scenario and call tools when needed, such as booking a dental appointment or ordering groceries. Six scenarios add up to 221 spoken turns with corrections, false memory traps, prompt injection and state tracking. An LLM judge grades each turn against a golden answer. | Everyday | Feb 2026 | 85.7 | Not in cards |
| AutoElicit An optimizer perturbs benign instructions to elicit unsafe behavior from computer-use agents. Reports elicitation success rate on transferred prompts. | Expert | Feb 2026 | | System cards only |
| Bio-Literature Search Level 2 Agents use search tools to compare abstracts from PubMedQA, including their sections, subject headings, and conclusions. Five task types require retrieving or combining evidence. | University | Feb 2026 | Not in cards | |
| BioMysteryBench Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. | Expert | Feb 2026 | 83.7% | In use |
| BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. | Expert | Feb 2026 | 87.7% | System cards only |
| Blicket Test Causal Reasoning Models experiment with a simulated detector to infer which objects activate it. The hub version rewards correct identification and informative experiments. | Everyday | Feb 2026 | | Not in cards |
| CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. | Everyday | Feb 2026 | Not in cards | |
| CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. | Expert | Feb 2026 | 25.8% | Fading |
| CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. | Expert | Feb 2026 | 74.8% | Fading |
| CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. | Expert | Feb 2026 | 72.2% | Fading |
| CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. | Expert | Feb 2026 | 52.3% | System cards only |
| ContextBench Coding agents resolve 1,136 issues from 66 repositories in eight programming languages, and the benchmark checks which code they look at along the way. Each task has human-annotated gold context, and the tracker scores context recall, precision and efficiency over the agent's trajectory. It adds these process measures to the usual issue-resolution result. | Expert | Feb 2026 | 60.6% | Not in cards |
| Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. | Expert | Feb 2026 | 16.4 | System cards only |
| DARE-Bench Agents write code to produce predictions from structured files. The benchmark checks predictions and adherence to modeling instructions. | University | Feb 2026 | 49.3% | Not in cards |
| Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. | Expert | Feb 2026 | | System cards only |
| DEEPSYNTH Agents search multiple websites and combine evidence to answer 120 research tasks across seven domains. They return structured JSON answers scored with precision, recall, F1, and exact match. A separate model judge assesses answers beyond exact matching. | Expert | Feb 2026 | 8.97% | Not in cards |
| DISBench Multimodal agents search raw visual histories over several steps to find target images from implicit context cues. | Everyday | Feb 2026 | 28.7% | Not in cards |
| DRACO 100 deep-research tasks from real Perplexity queries across 10 domains, graded on about 40 expert rubric criteria each. | Expert | Feb 2026 | 88.6% | In use |
| DraftNEPABench Agents draft sections of environmental impact statements using technical and regulatory source documents. Model judges score the drafts against rubrics, and specialists assess a sample. | Expert | Feb 2026 | 2.41 | Not in cards |
| EnterpriseBench: CoreCraft Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy. | Expert | Feb 2026 | 77.4% | In use |
| EnterpriseRAG-Bench Retrieval systems and agents answer 500 questions over a synthetic company's internal documents, slightly over 500,000 of them. | Expert | Feb 2026 | 91.5% | Not in cards |
| EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. | Expert | Feb 2026 | 51% | Not in cards |
| FeatureBench Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically. | Expert | Feb 2026 | 60.2% | Not in cards |
| FeatureBench Lite The public 30-task Lite leaderboard ranks passed-test percentages and reports fully resolved-task percentages separately. | Expert | Feb 2026 | 60.2% | Not in cards |
| Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. | Expert | Feb 2026 | | Fading |
| FPL 2026 Agentic FPGA Backend Optimization Competition Teams improve the maximum clock frequency of routed FPGA designs while preserving behavior. The organizers rank teams across seven designs after accounting for runtime and model cost. | Expert | Feb 2026 | Not in cards | |
| Game Arena Chess Openings Language models play chess against each other from varied two-move openings sampled from 20 popular human openings. The board is given as text. | Expert | Feb 2026 | | Not in cards |
| Game Arena Four in a Row Language models play a Connect Four style game and try to line up four of their tokens before the opponent does. | Everyday | Feb 2026 | | Not in cards |
| Game Arena Poker Language models play heads-up no-limit Texas Hold'em against each other. Models are ranked by mean big blinds won per 100 hands. | Expert | Feb 2026 | 30.3 | Not in cards |
| Game Arena Werewolf Teams of language models play the social deduction game Werewolf in natural language. Villagers must find the hidden werewolves, who try to deceive them. | Expert | Feb 2026 | 0 | Not in cards |
| GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result. | University | Feb 2026 | 70.6% | Not in cards |
| GDPval-MM Agents produce office deliverables for GDPval tasks in a MiniMax agent framework. A judge model compares full trajectories pairwise and reports a win rate. | Expert | Feb 2026 | 73.5% | Fading |
| IFC-Bench v2 Agents inspect IFC files and answer questions about building elements and their relationships. A model grades each answer against a reference. | University | Feb 2026 | 65.8% | Not in cards |
| LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. | Expert | Feb 2026 | 60.9% | In use |
| LABBench2 LitQA3 Models answer open-response questions whose answers appear once in recent biology papers. The LitQA3 subtask of LABBench2. | Expert | Feb 2026 | 85.1% | Not in cards |
| LABBench2 PatentQA Models locate and interpret evidence in patent documents. | Expert | Feb 2026 | 90.9% | Not in cards |
| LHAW Agents face 285 underspecified variants of long-horizon tasks with goals, constraints, inputs or context removed. Scored on strategic clarification and task success. | Expert | Feb 2026 | 68.4% | Not in cards |
| LLM-WikiRace Models play the Wikipedia game: starting from one article, they follow one hyperlink at a time to reach a target article. At each step the model sees only the links on the current page, and a game is lost after 30 steps. Three difficulty levels are set by the shortest path between the two pages. Scores are the share of games won at each level. | Everyday | Feb 2026 | | Not in cards |
| LOCA-Bench Agentic tasks whose environment state is scaled to grow context length while task semantics stay fixed. Success rate versus context length. | University | Feb 2026 | 21.3% | In use |
| MemGUI-Bench Agents remember details while working across Android apps. Tasks include recording audio and creating calendar events. A model checks the actions and results across repeated attempts. | Everyday | Feb 2026 | | Not in cards |
| MEWC Models solve 179 Microsoft Excel World Championship problems from 2021 to 2026 with Excel tools. Output cells are compared one by one with answer cells. | Competition | Feb 2026 | | Fading |
| MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. | Everyday | Feb 2026 | | Fading |
| Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. | Research frontier | Feb 2026 | 76% | System cards only |
| OdysseyArena Agents explore interactive environments built from four transition primitives and must work out the hidden dynamics over long horizons. OdysseyArena-Lite has 120 standard tasks. | Expert | Feb 2026 | | Not in cards |
| OdysseyArena AI Trading Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials. | Expert | Feb 2026 | 23.3% | Not in cards |
| OdysseyArena Energy Dispatch Agents manage energy dispatch under hidden rules. The score is average task success over four trials. | Expert | Feb 2026 | 23.3% | Not in cards |
| OdysseyArena Repo System Agents operate repositories with changing hidden rules. The score is average task success over four trials. | Expert | Feb 2026 | 62.5% | Not in cards |
| OdysseyArena Turn on Lights Agents learn hidden switches and turn on lights. The score is average task success over four trials. | Expert | Feb 2026 | 28.3% | Not in cards |
| OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. | University | Feb 2026 | | Fading |
| OpenEarthAgent Agents use tools such as QGIS to measure features in satellite images and maps. The authors grade tool decisions and final answers. | University | Feb 2026 | 44.5% | Not in cards |
| OpenThoughts-TBLite Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates. | Expert | Feb 2026 | 75.1% | Not in cards |
| Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. | Expert | Feb 2026 | 9.40% | System cards only |
| PinchBench Real-world tasks (scheduling, email triage, coding, research, files) for an LLM driving an OpenClaw agent. Graded by automated checks and LLM judge. | Everyday | Feb 2026 | | In use |
| ProxyWar Models write bots for nine games. The harness tests each bot and runs tournaments to measure wins, errors, and decision time. | University | Feb 2026 | | Not in cards |
| Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. | Expert | Feb 2026 | 70% | System cards only |
| RISE Agents answer real questions from human experts that need multi-step search and interaction with information-dense web pages. MiniMax has not published the scoring method. | Expert | Feb 2026 | | Fading |
| RuneBench Coding agents play RuneScape on an emulated game server by writing TypeScript against the rs-sdk library, with game-wiki notes for reference. Tasks train one skill or gather gold within 15 or 30 minutes. Skill tasks are scored by the peak XP rate in any 15-second window, and a verifier reads the final game state. | Everyday | Feb 2026 | 7.85 | Not in cards |
| SalesBench (Insurance Sales) Agents find insurance leads, quote plans, and schedule callbacks in a simulated sales pipeline. The evaluator rewards converted monthly premiums and checks compliance with calling restrictions. | University | Feb 2026 | Not in cards | |
| scBench (LatchBio SingleCellBench) 195 verifiable single-cell RNA-seq analysis problems. Each pairs a data snapshot with a task and a deterministic grader. | Expert | Feb 2026 | 63.9% | System cards only |
| scBench (original study) Agents complete 394 tasks in single-cell data analysis. The authors compare their answers with results from the original analyses. | Expert | Feb 2026 | 52.8% | Not in cards |
| SDR-Bench Models personalize sales pitches using customer research. A judge compares each pitch with points from successful human outreach. | University | Feb 2026 | 55.8% | Not in cards |
| SimulCost Agents tune physics simulations to meet an accuracy target. The authors measure success and computational cost for an initial guess and for repeated attempts with feedback. | Expert | Feb 2026 | 80.8% | Not in cards |
| SkillInject Agents use a skill file that carries an adversarial instruction while doing a legitimate task. Reports how often the agent follows the injection. | Expert | Feb 2026 | | System cards only |
| SkillsBench 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. | Expert | Feb 2026 | 66.5% | In use |
| SOP-Bench Agents follow expert-written standard operating procedures across 12 industrial domains, using executable tool interfaces over more than 2,000 tasks with 10 to 50 decision points. | Expert | Feb 2026 | Not in cards | |
| Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. | Expert | Feb 2026 | 79% | System cards only |
| Swarm Bench Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. | Expert | Feb 2026 | 76.3% | System cards only |
| SWE-rebench V2 Agents repair real repository issues using executable tests. The authors compare models on 300 tasks across five languages. Harbor also ports 51 validated Clojure tasks. | Expert | Feb 2026 | 25.2% | Not in cards |
| Terminal-Bench 2.0 Verified (Z.ai) Agents do the Terminal-Bench 2.0 tasks after Z.ai fixed environments for 89 tasks and instructions for 11. A further 2026-08-18 round fixed tests and instructions in 6 more tasks. | Expert | Feb 2026 | 61.8% | Not in cards |
| Vals Finance Agent Agents answer 537 financial analyst questions on public company SEC filings with search and filing tools. A three-vote GPT-5.2 judge grades each final answer. | Expert | Feb 2026 | 64.4% | Fading |
| VDR-Bench Models answer 2,000 visual questions that require finding entities in images and researching related facts. A Qwen3-VL judge checks final answers against references. Entity recall separately measures whether searches retrieve the relevant entities. | University | Feb 2026 | 27.2% | Not in cards |
| VenusBench-Mobile Agents use Android apps such as VLC, Markor, and Pro Expense. A model judges 90 primary tasks. The other tasks use programmatic checks. | Everyday | Feb 2026 | 24.8% | Not in cards |
| VeRO-Bench Coding agents improve other agents by editing their prompts, tools, and workflows. Each task has a reference evaluation for comparing changes. | Expert | Feb 2026 | 0.52 | Not in cards |
| VIBE-Pro (MiniMax) Agents build full-stack projects for Web, Android, iOS and Windows, including server APIs and databases. Claude Code based verifiers check interaction logic and visual output. | Expert | Feb 2026 | | Fading |
| Wiki Live Challenge Deep research agents write articles on topics of 100 recent Wikipedia Good Articles, which serve as expert references. Scored on 39 writing criteria and factual verifiability. | Expert | Feb 2026 | 53.6 | Not in cards |
| Windsurf Arena Mode Leaderboard Developers compare two model responses to the same coding request inside the editor. Their choices determine model Elo ratings. | University | Feb 2026 | 1,098 | Not in cards |
| WordleArenas Models guess a five-letter word from colored clues. Each arena changes how many clues give false feedback after a guess, from zero to five. | Everyday | Feb 2026 | 100% | Not in cards |
| YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. | University | Feb 2026 | $3,445,495 | In use |
| ABC-Bench Coding agents take a real backend repository, edit the code, set up the environment and deploy the service in a container. The 224 tasks come from 127 open-source repositories in 8 languages and 19 frameworks. A task passes only when the running service passes external end-to-end API tests. | Expert | Jan 2026 | 63.2% | Not in cards |
| AgencyBench Agents complete 138 long tasks drawn from daily AI use, grouped into 32 scenarios that cover six agentic capabilities such as coding, front-end work, game development and research. A typical task takes about 90 tool calls, 1 million tokens and hours of work. A simulated user gives feedback during the run, and a Docker sandbox checks the deliverables against task rubrics. | Expert | Jan 2026 | 56.5% | Not in cards |
| Agent Poker Bench Models play 10-seat no-limit Texas Hold'em tournaments against each other. Chip counts and elimination order feed TrueSkill ratings. | Expert | Jan 2026 | 1,132 | Not in cards |
| AI Office Benchmark (Moonshot) Agents complete end-to-end office tasks and produce documents, spreadsheets, PDFs and slide decks. Experts rate output quality. | Expert | Jan 2026 | Not in cards | |
| APEX-Agents 480 long-horizon tasks in banking, consulting and law worlds with docs, sheets, email and chat. Rubric-graded, Pass@1. | Expert | Jan 2026 | 67.2% | In use |
| APEX-SWE Software-engineering productivity tasks spanning integration work and observability, such as diagnosing failures from telemetry. | Expert | Jan 2026 | 67.6% | In use |
| Artificial Analysis Intelligence Index v4.0 Version 4.0 of Artificial Analysis's index, ten evaluations in four equal categories that brought in real-world knowledge work and hallucination tests. | Jan 2026 | 60.2 | Fading | |
| Bash Agent Evals Agents query GitHub issues and pull requests through bash, SQL, or a combination. Braintrust scores answers against the dataset. | University | Jan 2026 | Not in cards | |
| BU Bench V2 Browser agents do real web tasks with the Browser Use library. A judge scores each run against a rubric. | Everyday | Jan 2026 | 80.6 | Not in cards |
| Buyout Game Eight language models play a money-driven elimination game with private transfers and a buyout endgame. | Everyday | Jan 2026 | 1975.4 | Not in cards |
| CAR-bench Agents act as an in-car voice assistant that talks with an LLM-simulated driver and uses 58 connected tools for navigation, productivity, charging and vehicle control under set policies. Besides ordinary tasks, hallucination tasks remove a needed tool or fact, and disambiguation tasks need a clarifying question or a lookup. Scores reward consistent success across repeated runs. | Everyday | Jan 2026 | 70% | Not in cards |
| CooperBench Two coding agents each get a different feature for the same repository and must coordinate so both changes work together. Over 600 tasks span 12 libraries in 4 languages. | Expert | Jan 2026 | 27.9% | Not in cards |
| DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. | Everyday | Jan 2026 | 58.9% | Fading |
| DeepResearch Bench II Deep research agents write reports for 132 research briefs in 22 domains, each derived from a review article they may not consult. A judge model checks 9,287 expert criteria. | Expert | Jan 2026 | 43.1% | Not in cards |
| DeepResearch Bench II (Gemini 2.5 Pro judge) Systems conduct deep research under the legacy Gemini 2.5 Pro evaluator. Scores remain separate from the current GPT-5.5 evaluator. | Expert | Jan 2026 | 45.4% | Not in cards |
| DeepResearchEval Deep research systems answer complex research tasks that require evidence from many web sources. A persona-driven pipeline writes the tasks and keeps only those that need outside search. A model judge scores each report on general and task-specific dimensions, and an agent fact-checks the report's statements with web search even when citations are missing. | Expert | Jan 2026 | Not in cards | |
| DeepResearchEval v1 Research agents prepare reports. The score is average report quality on the original evaluation tasks. | Expert | Jan 2026 | Not in cards | |
| DeepSearchQA Questions that need multi-step search and synthesis. Final answers graded against references. | Expert | Jan 2026 | 95% | In use |
| DevOps-Gym Agents build, monitor, repair and test Java and Go projects. The Prime port converts tasks to single-turn questions with proxy grading. | University | Jan 2026 | 27.5% | Not in cards |
| General Agent Benchmark (Moonshot) Agents complete multi-step production workflows. Experts compare their work with human expert performance. | Expert | Jan 2026 | Not in cards | |
| KernelBench-Verified Language models generate CUDA kernels that must match PyTorch outputs and improve runtime. The verifier checks correctness with robust performance baselines. | Research frontier | Jan 2026 | Not in cards | |
| KradleVerse Agent harnesses compete in a live Minecraft arena. | Everyday | Jan 2026 | Not in cards | |
| LiveNewsBench Release 2 Models search for answers to questions about recent news. The test limits searches and page visits and compares answers with curated references. | Everyday | Jan 2026 | | Not in cards |
| mage-bench Models play Magic: The Gathering in a benchmark that compares strategic play across games. | Expert | Jan 2026 | 1617.0 | Not in cards |
| MegaGem Three models bid in sealed auctions with private cards. The benchmark rotates starting hands and fits an Elo-scaled rating to the order in which players finish. | Everyday | Jan 2026 | Not in cards | |
| METR Time Horizon 1.1 Agents attempt 228 software and research tasks timed against skilled humans, run in Inspect. The score is the human task length at which the agent succeeds half the time. | Research frontier | Jan 2026 | 17.4 h | Not in cards |
| MineBench Models turn a prompt into a Minecraft-style voxel build as raw 3D coordinates or tool calls. Users compare builds blind, and votes become rankings. | Everyday | Jan 2026 | 2,169 | Not in cards |
| MK-Bench Agents schedule jobs when machines break down or orders arrive or disappear. The grader compares each completion time with the best known schedule. | University | Jan 2026 | | Not in cards |
| NBABench (Thierry Damiba) Models search NBA player statistics from two seasons and answer 102 generated questions. The reference answers come directly from player records. | Everyday | Jan 2026 | Not in cards | |
| OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported. | Expert | Jan 2026 | 81% | Not in cards |
| OTelBench Coding agents add OpenTelemetry tracing, metrics and logging to codebases in several languages. Tests check the instrumentation. | Expert | Jan 2026 | 29% | Not in cards |
| OverSearchQA Search-augmented models decide when to abstain from retrieval on unknown, false-premise and underspecified questions. | University | Jan 2026 | Not in cards | |
| PaperScope Agents search multiple scientific papers to compare figures, summarize results, and solve problems. Each question can require evidence from several documents. | Expert | Jan 2026 | 41.0 | Not in cards |
| Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. | Expert | Jan 2026 | Not in cards | |
| Prediction Arena Agents research events and trade contracts on Kalshi and Polymarket. The leaderboard ranks accounts by cash plus the market value of open positions. | University | Jan 2026 | $138,945 | Not in cards |
| prinzbench Models research fact patterns under U.S. statutes and regulations and hunt for obscure public information online. 33 private questions run three times give a score out of 99. | Expert | Jan 2026 | 91.0 | Not in cards |
| ProofBench Models write Lean 4 proofs of graduate-level theorems, with Mathlib search and code tools for up to 40 turns. Scored by the share that compile. | Expert | Jan 2026 | 78% | Not in cards |
| Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. | University | Jan 2026 | | Fading |
| RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. | Everyday | Jan 2026 | 76.3% | Not in cards |
| RWSearch Agents search the live web to answer 200 expert-written Chinese questions that need many retrieval steps, filtering and arithmetic. Each question has one objective answer. | Expert | Jan 2026 | 82% | Fading |
| ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. | Expert | Jan 2026 | | Fading |
| Snorkel Agentic Coding Agents complete 100 multi-step coding tasks in sandboxed containers, spread evenly over four difficulty tiers. Unit tests and rubrics grade both outputs and the agent trajectory. | Expert | Jan 2026 | 65.2% | Not in cards |
| SokoBench Models solve Sokoban puzzles that require long-horizon planning and persistent state tracking. | Everyday | Jan 2026 | Not in cards | |
| Spyfall study Models play Spyfall while inferring hidden locations and managing deceptive dialogue. | Everyday | Jan 2026 | | Not in cards |
| Strategy Bench Models compete in six social deduction and strategy game environments. | Everyday | Jan 2026 | Not in cards | |
| SWE-gen JS Agents fix issues drawn from 1,000 merged pull requests across 30 JavaScript and TypeScript repositories. Each task includes tests that fail before the reference fix. | Expert | Jan 2026 | Not in cards | |
| tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Everyday | Jan 2026 | | Fading |
| TaxoBench Agents find research papers and organize them into a taxonomy. The evaluator compares their groupings and hierarchy with taxonomies from 72 published surveys. | Expert | Jan 2026 | Not in cards | |
| TetrisBench Models play Tetris head-to-head in the original benchmark. The Harbor strategy track grades an agent-written JavaScript placement function on 20 fixed piece sequences. | Everyday | Jan 2026 | 66.7% | Not in cards |
| TradeRank Arena Models choose positions in cryptocurrencies and US stocks using shared market data and simulated capital. The leaderboard ranks portfolios by return. | University | Jan 2026 | 0% | Not in cards |
| Video Browser Showdown 2026 marine video search Participants use interactive retrieval systems to find five underwater clips after seeing each target. Each search has a five-minute limit. | Everyday | Jan 2026 | Not in cards | |
| VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. | University | Jan 2026 | | In use |
| VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Everyday | Jan 2026 | | Fading |
| Vmax Tasks Agents fix JavaScript bugs from projects including Vue.js, Redux, Docusaurus, and Chalk. Harbor lists 1,043 tasks with bug reports and expected behavior. | Expert | Jan 2026 | Not in cards | |
| AA-Briefcase Artificial Analysis knowledge-work evaluation of professional deliverables. Models ranked by Elo. | Expert | 2026 | 1,720 | In use |
| LatchBio Capabilities v1.0 LatchBio cross-benchmark suite of agentic biological data analysis (transcriptomics, epigenomics, variants, pharmacology) with deterministic graders. | Expert | 2026 | 58% | System cards only |
| Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. | Expert | Dec 2025 | 100% | System cards only |
| AgentIF-OneDay Agents complete 104 tasks that stand for a full day of human work, life and study. Tasks include tedious workflows, inferring rules from examples and changing requirements. | Expert | Dec 2025 | 62.6% | Not in cards |
| AI Consumer Index Models with web search handle everyday consumer requests in shopping, food, gaming and DIY. Prompt-specific rubrics with hurdle and grounding criteria grade each answer. | Everyday | Dec 2025 | 56.1% | Not in cards |
| AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. | Expert | Dec 2025 | | Fading |
| AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. | Expert | Dec 2025 | 44% | Not in cards |
| AMUSE Multimodal models reason over multi-speaker audio-visual interactions in zero-shot, guided and agentic modes across six task families. | Expert | Dec 2025 | Not in cards | |
| App-Bench Coding agents build six full-stack web applications from a single prompt. Two developers grade implemented features against 151 rubric items. The score uses each tool’s best result from three attempts per application. | University | Dec 2025 | 67.5% | Not in cards |
| Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. | Expert | Dec 2025 | 90.9% | In use |
| BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. | Expert | Dec 2025 | 99.1% | System cards only |
| BinaryAudit Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software. | Expert | Dec 2025 | 49.1% | Not in cards |
| CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. | Research frontier | Dec 2025 | 8.40 | Not in cards |
| CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. | Research frontier | Dec 2025 | | Not in cards |
| CNFinBench Models answer financial questions and carry out financial workflows. The benchmark includes tasks on expertise, autonomy, and integrity. | Expert | Dec 2025 | | Not in cards |
| CubeBench Agents solve Rubik's Cube tasks across symbolic, visual and partially observed settings to measure spatial reasoning and long-horizon state tracking. | Expert | Dec 2025 | Not in cards | |
| CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. | Expert | Dec 2025 | | In use |
| DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. | Expert | Dec 2025 | 47% | Not in cards |
| FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. | University | Dec 2025 | | Fading |
| FACTS Search Models answer information-seeking questions with a web search tool (the Brave Search API) and must combine what they find correctly. | University | Dec 2025 | | Not in cards |
| Finch (FinWorkBench) Agent completes 172 enterprise finance and accounting spreadsheet workflows (384 tasks) from real data. Human plus calibrated LLM-judge grading. | Expert | Dec 2025 | 43% | In use |
| FrontierCS 1.1 Agents solve 179 algorithmic tasks under revised scoring that rewards solutions beyond the reference. The score is the official average task score. | Research frontier | Dec 2025 | 56.5 | Not in cards |
| GDPval-AA Agents complete GDPval occupational tasks. Artificial Analysis compares the outputs in pairs and ranks models by Elo. | Expert | Dec 2025 | 1,932 | Fading |
| LocalSearchBench Agents search merchant listings to answer 900 questions about local services across nine Chinese cities. A model judges each answer against a reference. | Everyday | Dec 2025 | | Not in cards |
| MCP-SafetyBench Agents complete tool-use tasks across five domains. The benchmark reports both task completion and resistance to adversarial inputs. | University | Dec 2025 | 21.2% | Not in cards |
| MedCalc-Bench Verified Models extract values from patient notes and compute results for 55 medical calculators. Verified release of MedCalc-Bench with corrected labels. | Expert | Dec 2025 | 86.2% | System cards only |
| MobileWorld Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. | Everyday | Dec 2025 | 55.3% | In use |
| NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. | Expert | Dec 2025 | | In use |
| OctoCodingbench Agents complete 72 coding tasks while following rules from system prompts, AGENTS.md files, skills, and memory. An LLM judge checks 2,422 pass or fail items. | University | Dec 2025 | 36.2% | Fading |
| Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. | Expert | Dec 2025 | | System cards only |
| Pieverse Agent Prediction Arena Agents trade event contracts on Polymarket with $1,000 each in real money. Pieverse tracks account value as market prices change. | University | Dec 2025 | $981 | Not in cards |
| PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. | Expert | Dec 2025 | 99.6% | System cards only |
| REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. | Everyday | Dec 2025 | 53.1% | Not in cards |
| SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. | Everyday | Dec 2025 | 57.8% | Not in cards |
| SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. | Expert | Dec 2025 | $548,400,000 | Not in cards |
| SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. | Expert | Dec 2025 | | In use |
| Solubility Optimization (anindya) Models choose chemical transformations to improve predicted solubility. The package includes a benchmark runner that measures improvement on evaluation examples. | University | Dec 2025 | Not in cards | |
| SpatialBench (LatchBio) Agentic spatial-transcriptomics data analysis problems with deterministic graders. Anthropic reports a 115-problem Verified variant. | Expert | Dec 2025 | 57.6% | System cards only |
| SpatialBench (original study) Agents complete 146 tasks in spatial transcriptomics. The authors compare their answers with results from the original analyses. | Expert | Dec 2025 | 38.4% | Not in cards |
| SpatialTree Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level. | Everyday | Dec 2025 | | System cards only |
| SpreadsheetBench Verified Models solve 400 real spreadsheet manipulation questions from Excel forums, an expert annotated subset of SpreadsheetBench. Each answer runs against several test spreadsheets. | University | Dec 2025 | | Fading |
| SusVibes Coding agents rebuild a feature in a real open-source Python project from a plain-language request. Each task comes from a historical vulnerability fix, so the original human version was insecure. Hidden functional tests and security tests check the result. Scores report the share of solutions that work and the share that are also secure. | Expert | Dec 2025 | 43.5% | Not in cards |
| SWE-EVO Coding agents implement release-note changes in seven mature Python projects, editing about 21 files per task. Large test suites check each of the 48 tasks. | Expert | Dec 2025 | 39.6% | Fading |
| Tau2 Bench Verified Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. | University | Dec 2025 | Not in cards | |
| tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. | Everyday | Dec 2025 | 82.7% | Fading |
| Terminal-Bench Pro Agents solve 400 terminal tasks across eight domains. Harbor distributes the 200 public tasks. Alibaba keeps the other 200 for private evaluation. | Expert | Dec 2025 | 51% | Not in cards |
| VIBE (MiniMax) Agent builds complete web, simulation, Android, iOS or backend apps from scratch. Agent verifiers check execution, interaction and visuals. | Expert | Dec 2025 | 90.7% | Fading |
| WebArena-Verified Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. | Everyday | Dec 2025 | | In use |
| WOLF Models play Werewolf to measure deception production and deception detection as separate abilities. | Everyday | Dec 2025 | Not in cards | |
| WorldTravel Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data. | Everyday | Dec 2025 | 32.7% | Fading |
| A.S.E 2.0 Models and agentic coding tools write code in real projects covering 29 CWE types. A mix of static analysis, tests and vulnerability proofs of concept checks security. | Expert | Nov 2025 | Not in cards | |
| AndroidDaily Agents complete 350 daily tasks in 94 real closed-source Android apps. A process-aware reviewer model checks each trajectory against written guidelines. | Everyday | Nov 2025 | | Fading |
| ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. | Expert | Nov 2025 | 66.8% | System cards only |
| Audio Reasoning Challenge 2026 Models answer questions about speech, music, and environmental audio and explain their reasoning. Incorrect answers score zero. A model judge grades correct answers for reasoning quality, with the middle three of five evaluation runs averaged. | Everyday | Nov 2025 | Not in cards | |
| Augmented Reality Patent Search Agents search 12,000 augmented and virtual reality patent applications to answer technical questions. A model judges each answer. | University | Nov 2025 | Not in cards | |
| Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. | Expert | Nov 2025 | 73.7% | System cards only |
| BrowseSafe-Bench Detectors and browser agents face prompt injections hidden in realistic HTML pages, across 11 attack types, nine placements and three writing styles. 3,680 test samples. | University | Nov 2025 | 86.3% | Not in cards |
| CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. | Expert | Nov 2025 | System cards only | |
| Code Arena Models act as coding agents that build web apps from user prompts, and users vote between two results. Votes yield Elo-style ratings. | Nov 2025 | 1,826 | In use | |
| CodeClash Models build and improve their own codebases over many rounds, then compete in arenas such as Halite, poker and BattleSnake. Results give Elo ratings. | Expert | Nov 2025 | 1,385 | Not in cards |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Expert | Nov 2025 | | System cards only |
| DeepAmbigQA Search-augmented models answer multi-hop questions that combine entity-name ambiguity with complete answer-set retrieval. | University | Nov 2025 | Not in cards | |
| Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. | Expert | Nov 2025 | 90% | System cards only |
| Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. | Everyday | Nov 2025 | | System cards only |
| Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. | Expert | Nov 2025 | 0% | System cards only |
| Mathematical Problem Repository for AlphaEvolve Agents generate programs that construct mathematical objects or improve known bounds. The repository documents 67 problems and provides verification code for many of them. | Research frontier | Nov 2025 | Not in cards | |
| MCP Atlas Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. | University | Nov 2025 | | In use |
| Mini Amusement Parks Agents build rides, hire staff, and manage an amusement park. The simulator scores the park’s final value. | Everyday | Nov 2025 | 371375.0 | Not in cards |
| Multi-agent search (Anthropic internal benchmark) Agents solve difficult information retrieval problems with web search, alone or as orchestrators that delegate to subagents. | Expert | Nov 2025 | 74.8% | System cards only |
| NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. | Expert | Nov 2025 | Not in cards | |
| OASIS (agent safety) Agents handle multi-turn requests with different levels of complexity. Evaluators measure the proportion of safe responses. | University | Nov 2025 | 95.2% | Not in cards |
| OSGym Infrastructure running 1,000+ parallel OS sandboxes for computer-use agent training and evaluation. Tasks span browsing, documents, SWE and multi-app work. | Everyday | Nov 2025 | Not in cards | |
| Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. | Expert | Nov 2025 | 0% | System cards only |
| Prompt injection robustness against adaptive attackers, coding (Anthropic) Coding agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. | Expert | Nov 2025 | 0% | System cards only |
| Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. | Expert | Nov 2025 | 0% | System cards only |
| PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. | Expert | Nov 2025 | 10.5% | System cards only |
| Real-Time Reasoning Gym Models act in games while token generation consumes part of the game clock. | Everyday | Nov 2025 | | Not in cards |
| ResearchRubrics 101 deep-research prompts with 2,593 expert-written weighted rubric criteria (including penalties). Reports graded for grounding, reasoning and clarity. | Expert | Nov 2025 | | In use |
| SWE-fficiency 498 tasks in nine Python data/ML/HPC repos: agent must speed up a real workload without breaking tests. Scored as speedup relative to expert patch. | Expert | Nov 2025 | 22.5% | In use |
| Tau2 Bench Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. | University | Nov 2025 | Not in cards | |
| Terminal-Bench 2.0 89 hard terminal tasks across 16 categories in containers. Hidden tests verify the final state. 5 trials per task. | Expert | Nov 2025 | 82.3% | In use |
| TIR-Bench Models solve 13 kinds of image tasks that need tool use on images, such as rotation, jigsaw, and maze. Most items are multiple choice. | Everyday | Nov 2025 | | Fading |
| Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. | Everyday | Nov 2025 | | In use |
| Vending-Bench Arena Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance. | Expert | Nov 2025 | | Not in cards |
| Vibe Code Bench v1 Agents build web applications from scratch from a natural-language spec in OpenHands. The built apps are tested against functional requirements. | Expert | Nov 2025 | 41.3% | Not in cards |
| WebTailBench Computer-use agents complete 609 hand-verified tasks on live websites across 11 categories, such as bookings, real estate and job search. A companion set of 111 harmful tasks tests refusals. | Everyday | Nov 2025 | 57.4% | Fading |
| WebTailBench-Refusals Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused. | Everyday | Nov 2025 | | System cards only |
| WISER-FindAll Entity discovery systems return all entities that match complex multi-criteria web queries. | Expert | Nov 2025 | 21% | Not in cards |
| AgentBench FC Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format. | Everyday | Oct 2025 | 58.9% | Not in cards |
| Alpha Arena Frontier models trade live markets with real money over a season, and the leaderboard ranks them by returns. | Expert | Oct 2025 | Not in cards | |
| Biomni-Eval1 Agents use biomedical data and tools to answer 433 questions across ten task types. Each task checks a structured answer against a reference. | Expert | Oct 2025 | Not in cards | |
| BuildArena Agents design machines in the Besiege simulator from written requirements. The simulator tests whether a vehicle reaches its target, how much weight a bridge supports, and how a lifting machine performs. | University | Oct 2025 | 45.3% | Not in cards |
| BuilderBench Agents learn to control a simulated robot that builds target block structures through physical interaction and exploration. | Expert | Oct 2025 | 98% | Not in cards |
| Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. | Everyday | Oct 2025 | 37% | Not in cards |
| CATArena Code agents write programs that play board and card games in repeated tournaments. Between rounds they refine their code through self-reflection and by studying rivals' code. Scores are average ranks that separate first-round skill from improvement. | Expert | Oct 2025 | 1,799 | Not in cards |
| CATArena Gomoku Code agents compete in Gomoku. The public competition board reports Elo Rating for each agent configuration. | Expert | Oct 2025 | 1,799 | Not in cards |
| CATArena OIBench-5 Code agents compete in OIBench-5. The public competition board reports Time AUC for each agent configuration. | Expert | Oct 2025 | 0.40 | Not in cards |
| CATArena Pommerman Code agents compete in Pommerman. The public competition board reports Elo Rating for each agent configuration. | Expert | Oct 2025 | Not in cards | |
| CATArena SWE-Perf-10 Code agents compete in SWE-Perf-10. The public competition board reports Time Ratio for each agent configuration. | Expert | Oct 2025 | 1.95× | Not in cards |
| CATArena Tetris Code agents compete in Tetris. The public competition board reports Score for each agent configuration. | Expert | Oct 2025 | Not in cards | |
| CATArena Texas Holdem Code agents compete in Texas Holdem. The public competition board reports Elo Rating for each agent configuration. | Expert | Oct 2025 | 1,463 | Not in cards |
| Cognition CodeSearch Eval Agents retrieve relevant files and line ranges for questions about real repositories. Cognition compares their answers with human labels using precision-weighted F scores. | University | Oct 2025 | 0.59 | Not in cards |
| Cursor Bench (2025 internal version) Agents answer real agent requests from Cursor engineers in Cursor's harness. Graders compare against hand-curated solutions and check use of the codebase's existing abstractions. | Expert | Oct 2025 | Fading | |
| CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. | Expert | Oct 2025 | Not in cards | |
| Dojo Mini Bench Agents complete tasks in LinkedIn, Linear and Gmail through Dojo. Each task has its own completion check. | Everyday | Oct 2025 | Not in cards | |
| DPAI Arena Spring Benchmark Coding agents implement changes in Java Spring repositories. DPAI Arena scores each run and publishes its duration. | Expert | Oct 2025 | 64.9 | Not in cards |
| FML-bench Agents propose and run experiments on eight machine learning research tasks. The authors measure task performance and how widely the proposals vary across iterations. | Expert | Oct 2025 | 0.50 | Not in cards |
| Fruit Box Models select rectangles whose entries sum to ten on a ten-by-seventeen grid. The evaluator compares the final score with an expert trajectory from the training split. | School | Oct 2025 | Not in cards | |
| Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. | Expert | Oct 2025 | | System cards only |
| ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. | Expert | Oct 2025 | | System cards only |
| LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. | Everyday | Oct 2025 | | Not in cards |
| LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. | Everyday | Oct 2025 | 53% | Not in cards |
| LiveResearchBench Deep research agents write cited reports for user-centric questions that need current information from many live web sources. Rubric-based judges score the reports. | Expert | Oct 2025 | 73.7 | Not in cards |
| Magentic Marketplace Agents search for sellers, exchange proposals and make purchases. The simulation measures how much value customers receive after paying for their purchases. | Everyday | Oct 2025 | Not in cards | |
| Mastermind (Prime Intellect) Models guess a hidden sequence and receive feedback after each turn. The default evaluation uses 50 games and rewards success, speed, and deductions. | Everyday | Oct 2025 | 1.62 | Not in cards |
| MEMTRACK Agents track changes across software platforms and use earlier information to complete tasks. | Everyday | Oct 2025 | 61% | Not in cards |
| Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. | Expert | Oct 2025 | | System cards only |
| Next.js Agent Evals Coding agents with their models do real Next.js tasks. The board reports success rate with and without an AGENTS.md file, plus duration and cost. | Expert | Oct 2025 | 100% | Not in cards |
| OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. | University | Oct 2025 | | Fading |
| OpenGameEval Agents write Luau scripts and edit objects in Roblox Studio. The original suite checks 47 changes to game mechanics and scenes. | University | Oct 2025 | | Not in cards |
| OSWorld-MCP Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps. | Everyday | Oct 2025 | 56.6% | Not in cards |
| OutboundEval Agents conduct outbound calls with simulated users. The benchmark scores compliance with the professional workflow and the quality of the interaction. | Expert | Oct 2025 | Not in cards | |
| Petri Open-source auditing tool where auditor agents probe target models in simulated scenarios and judges score misaligned behaviors. | Expert | Oct 2025 | System cards only | |
| PRDBench Code agents build 50 real-world Python projects across 20 domains from product requirement documents. A trained judge agent checks each project against structured criteria. | Expert | Oct 2025 | 69% | Not in cards |
| Prophet Arena Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return. | Expert | Oct 2025 | 86.2 | In use |
| RECODE-H Agents implement research code from papers and repositories. They can request feedback from a simulated collaborator before their code runs against tests. | Expert | Oct 2025 | 84.3% | Not in cards |
| Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. | Expert | Oct 2025 | 41.3% | In use |
| ReplicationBench Agents reproduce computational results from published astrophysics papers. Evaluators check their numerical answers and assess whether their methods follow the original work. | Research frontier | Oct 2025 | 22% | Not in cards |
| RoboChallenge Fine-tuned policies run 10 real rollouts on each of 30 tabletop tasks across four robot types. Scored by success rate and progress. | Everyday | Oct 2025 | 42.7% | Not in cards |
| Sherlock Holmes Q&A Models search 55 Sherlock Holmes stories and answer 140 literary comprehension questions. A model judge checks their answers. | School | Oct 2025 | 100.0 | Not in cards |
| SQL-API-Bench Agents answer questions that require both database queries and API calls. The evaluator checks answers across two mixed database and API benchmark settings. | Expert | Oct 2025 | Not in cards | |
| StockBench Agents trade stocks over a fixed evaluation window. StockBench ranks models by the mean of final return, maximum drawdown and Sortino ratio. The score is the board's rank, so a lower score is better. | Expert | Oct 2025 | 1.00 | Not in cards |
| Toolathlon 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. | University | Oct 2025 | | In use |
| USIM underwater robot tasks Agents follow instructions to navigate, grasp, transport, and track objects underwater. The simulator measures task completion across 700 trials. | Everyday | Oct 2025 | 37.6% | Not in cards |
| Vals Index v1 Vals AI's first index of agentic work in finance, law and coding, with each sector weighted by its share of U.S. GDP. | Oct 2025 | 72.2% | System cards only | |
| Vals Multimodal Index v1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Expert | Oct 2025 | 71.1% | Not in cards |
| VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. | University | Oct 2025 | | Not in cards |
| WARC-Bench Screenshot-only agents do short web subtasks, such as date pickers and scroll containers, on real sites replayed from Web Archive snapshots. A program checks the end state. | Everyday | Oct 2025 | 64.8% | Not in cards |
| World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. | Everyday | Oct 2025 | 64.8% | Not in cards |
| xbench-DeepSearch-2510 Agents answer 100 new Chinese deep search questions that need many web searches and cross-checks. Answers are matched to a key. | Expert | Oct 2025 | | Fading |
| AI Spreadsheet Benchmark Spreadsheet assistants complete 53 prompts for analysis, charts, formulas, and workbook management. Checks measure task success and whether outputs update when source data changes. The evaluation reports pass@1, pass@3, dynamic output rate, and latency. | University | Sep 2025 | Not in cards | |
| Artificial Analysis Intelligence Index v3 Version 3 of Artificial Analysis's index, ten evaluations that added agentic terminal and tool-use tasks to the knowledge, math and coding tests. | Sep 2025 | | Dropped | |
| Automotive-ENV Agents operate climate controls in an Android Automotive OS simulator. A validator checks the screen and vehicle signals after requests such as turning on the defroster. | Everyday | Sep 2025 | 49.3% | Not in cards |
| BEHAVIOR Challenge 2025 50 long-horizon household activities on a Galaxea R1 Pro in OmniGibson, trained from 10,000 demos. Scored by Q-score and success. | Everyday | Sep 2025 | Fading | |
| BioML-bench Agents build and evaluate machine learning systems for biomedical experiments spanning proteins, drugs, single cells, imaging, and biomarkers. Task-specific metrics compare submissions with experimental reference results. | Expert | Sep 2025 | Not in cards | |
| Blackjack Basic Strategy Models play blackjack hands or select a single move from a dealt position. The grader measures the expected value of each decision against basic strategy. | Everyday | Sep 2025 | Not in cards | |
| BlueBench Agents solve enterprise tasks across knowledge, coding, security and business workflows. The benchmark aggregates capability scores from multiple public task suites. | Expert | Sep 2025 | 64.6 | Not in cards |
| Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. | Expert | Sep 2025 | 51.3% | Dropped |
| CTF-Archive Agents solve capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. | Expert | Sep 2025 | Not in cards | |
| DeepResearch Arena Deep research agents get research tasks generated from academic seminar transcripts, which reflect expert discussion and reduce data leakage. | Expert | Sep 2025 | 3.96 | Not in cards |
| DRBench Agents investigate open-ended business questions across enterprise applications and produce reports. The evaluator checks whether the reports recover relevant information and support their claims. | University | Sep 2025 | 63.8 | Not in cards |
| Earth-Bench Agents analyze satellite observations with Earth science tools. The evaluator checks the final answer and compares tool calls with a reference sequence. | University | Sep 2025 | 66% | Not in cards |
| FinSearchComp Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. | Expert | Sep 2025 | | In use |
| FinSearchComp Global Agents search for financial information about global assets across three task types. The score is the published average accuracy for the Global subset. | Expert | Sep 2025 | | Dropped |
| FinSearchComp Greater China Agents search for financial information about Greater China assets across three task types. The score is the published average accuracy for the Greater China subset. | Expert | Sep 2025 | Not in cards | |
| ICPC World Finals 2025 Models solve the 12 problems of the 2025 ICPC World Finals under contest time and attempt limits. Scored by problems solved. | Competition | Sep 2025 | | Fading |
| IMProofBench Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically. | Research frontier | Sep 2025 | 49.3% | Not in cards |
| Lights Out 3×3 Models toggle lights on a three-by-three grid until every light is off. The dataset reserves 30 boards for testing and rewards efficient solutions. | School | Sep 2025 | Not in cards | |
| LiveNewsBench Models search for answers to questions about recent news. The test limits searches and page visits and compares answers with curated references. | Everyday | Sep 2025 | 21.5% | Not in cards |
| Malicious use of Claude Code (Anthropic) Agents in Claude Code receive malicious and dual-use coding requests. Score is the refusal rate on malicious requests. | Expert | Sep 2025 | 96.7% | System cards only |
| MAPP (Multi-Agent Path Planning) Models move several agents across a grid without collisions. The environment tracks how many agents reach their goals and how closely their routes follow shortest paths. | School | Sep 2025 | 70.8% | Not in cards |
| Mastermind Information Gain Models choose a Mastermind move from earlier guesses and feedback. The evaluator measures information gain for individual moves and success over complete games. | School | Sep 2025 | Not in cards | |
| MCP prompt injection evaluation (Anthropic) Agents use simulated email, Slack and document servers over MCP where content carries injected instructions. Score is the attack prevention rate. | Expert | Sep 2025 | 92.5% | System cards only |
| MCPMark Agent stress-tests realistic MCP tool use (GitHub, Notion, Postgres, browser). Graded by programmatic verification scripts. | University | Sep 2025 | 94.5% | In use |
| Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. | Everyday | Sep 2025 | 42.1% | Fading |
| Multi-Turn Sudoku (m8ngotree) Models fill a nine-by-nine Sudoku board one move at a time. The generator checks for a unique solution, and the scorer rewards valid moves and completion. | School | Sep 2025 | Not in cards | |
| OceanGym Agents interpret underwater images and sonar, then navigate simulated search and inspection tasks. The simulator scores how closely they reach target locations. | University | Sep 2025 | 14.6 | Not in cards |
| OOLONG Models aggregate information across long collections of text. The hub port implements the synthetic and real datasets with a context-controlled recursive language model interface. | University | Sep 2025 | | Not in cards |
| Project Euler (MathArena) Models solve recent Project Euler problems that need both mathematical insight and code. MathArena adds problems as Project Euler releases them, starting from problem 943. | Expert | Sep 2025 | | Not in cards |
| Red Team Defense Simulator Models respond to an adaptive adversary. The evaluation measures successful defense, helpfulness and false refusals. | University | Sep 2025 | 85% | Not in cards |
| Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. | Expert | Sep 2025 | System cards only | |
| robust-kbench Agents write CUDA kernels that are tested across several weight initializations and input shapes, so shape and magnitude exploits fail. Correctness checks and profiling score each kernel. | Expert | Sep 2025 | Not in cards | |
| SCUBA Agents work in a Salesforce sandbox as administrators, sales representatives, or service agents. Evaluators check milestones such as creating accounts and routing support cases. | University | Sep 2025 | 53.9% | Not in cards |
| Security Configuration Verification Agents audit Kubernetes and Terraform configurations, identify security violations and submit fixes. The grader weights detected violations by severity and validates patches. | University | Sep 2025 | Not in cards | |
| SWE-Bench Pro 1,865 long-horizon issue-resolution tasks from 41 repos in four languages. Agent patch must pass hidden tests. Pass@1. | Expert | Sep 2025 | | In use |
| SWE-Bench Pro Private Agents resolve held-out software issues from the private SWE-Bench Pro split. The leaderboard reports the percentage of resolved tasks. | Expert | Sep 2025 | | Not in cards |
| Terminal-Bench Hard Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. | Expert | Sep 2025 | | In use |
| Tool use prompt injection evaluation (Anthropic) Agents run bash commands where files, scripts or command outputs carry injected instructions. Score is the attack prevention rate. | Expert | Sep 2025 | 96% | System cards only |
| UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. | Everyday | Sep 2025 | | Dropped |
| Valyrian Games Coding Challenge Models create coding problems with integer answers and repeatedly solve their own problems to qualify. Tournament entrants solve one another’s problems and receive TrueSkill ratings. | Competition | Sep 2025 | 30 | Not in cards |
| VisualProbe Models search high-resolution images for small targets to answer questions, often over many crop and zoom turns. Answers are checked against a key. | Everyday | Sep 2025 | | Fading |
| VITA-Bench Agents handle multi-turn requests in simulated food delivery, in-store, and travel services with 66 tools. A rubric-based judge scores task success. | Everyday | Sep 2025 | | Fading |
| VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. | Everyday | Sep 2025 | | Not in cards |
| X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. | Expert | Sep 2025 | 58% | Dropped |
| AI FPL Models manage Fantasy Premier League squads using the same weekly prompt. Their transfers, captain choices, and starting lineups earn points from real matches. | Everyday | Aug 2025 | Not in cards | |
| Android Agent Arena (100 tasks) Agents use 20 Android apps to find information and perform actions. A vision model checks whether the agent reached the required intermediate states and completed the task. | Everyday | Aug 2025 | | Not in cards |
| AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. | Expert | Aug 2025 | 58% | Not in cards |
| BalatroBench Models play the deck-building roguelike Balatro through an API bot. The leaderboard compares how far each model and strategy gets in a run. | Everyday | Aug 2025 | | Not in cards |
| Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. | Everyday | Aug 2025 | 0.33% | In use |
| Brokk Power Ranking Models solve 181 real tickets from a 1.1 million line Rust codebase. Model judges compare solutions in pairs, and a Bradley-Terry fit gives each model a strength score. | Expert | Aug 2025 | | Not in cards |
| BrowseComp-Plus Deep research agents answer 830 BrowseComp questions by searching a fixed corpus of about 100,000 human-verified documents. | Expert | Aug 2025 | 70.1% | In use |
| BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. | Expert | Aug 2025 | | Fading |
| Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. | Expert | Aug 2025 | 99% | System cards only |
| Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. | Expert | Aug 2025 | 1.30% | System cards only |
| CompileBench Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code. | Expert | Aug 2025 | 100% | Not in cards |
| ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. | Everyday | Aug 2025 | 66.5% | Dropped |
| CUA-SHADE-Arena Monitors review computer-use agent activity in simulated workplace tasks. The evaluation measures how often they flag unsafe behavior at a fixed false positive rate. | University | Aug 2025 | Not in cards | |
| DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. | Everyday | Aug 2025 | 39% | Not in cards |
| FutureX Agents predict upcoming events collected daily from 195 websites in 11 domains, within a one-week window. Answers are scored once real results are crawled, with harder tiers weighted more. | Expert | Aug 2025 | 27.9 | Not in cards |
| Game Arena Chess Language models play full chess games against each other with the board given as FEN and PGN text. Ratings come from many games between each pair. | Expert | Aug 2025 | | Not in cards |
| GEN-Bench Agents select job operations as conditions change in a simulated factory. The benchmark compares schedule completion times with the best known result for each instance. | University | Aug 2025 | | Not in cards |
| GitTaskBench Code agents solve 54 real tasks across 7 modalities and 7 domains by using a given GitHub repository, from understanding and setup to running it. | Expert | Aug 2025 | 48.1% | Not in cards |
| Hallucinations on production-representative prompts Models answer prompts representative of ChatGPT production traffic. A grader with web access scores incorrect claims and responses with major errors. | Everyday | Aug 2025 | 3.10% | System cards only |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | Expert | Aug 2025 | 16% | Fading |
| IOI (Vals) Agents solve International Olympiad in Informatics problems from 2024 and 2025 in C++. A submission tool modeled on the contest grades up to 50 tries per problem by subtask. | Competition | Aug 2025 | 91.7% | Not in cards |
| Kaggle Game Arena Language models play each other in board, card and word games through a text harness. One leaderboard ranks models across all Game Arena games. | Expert | Aug 2025 | | Not in cards |
| LiveDRBench Agents perform 100 deep research tasks and return key claims discovered during search. Scores use claim-level F1. | Expert | Aug 2025 | 55.5% | Not in cards |
| LiveTradeBench Agents allocate simulated portfolios across US securities and Polymarket contracts using streamed prices and news. The researchers track portfolio returns over successive trading days. | University | Aug 2025 | 12.1% | Not in cards |
| LLM Chess Arena (Louis Guichard) Models play each other from a text diagram, piece locations, and recent moves. A model forfeits after three failed attempts to make a legal move. | School | Aug 2025 | | Not in cards |
| M3-Bench Multimodal agents answer questions about long videos while using long-term memory and multimodal retrieval. | Expert | Aug 2025 | Not in cards | |
| MCP-Bench Agents solve multi-step tasks using 28 live MCP servers with 250 tools in areas such as finance, travel, scientific computing and academic search. Scoring covers tool use, coordination and planning. | Expert | Aug 2025 | 0.75 | Not in cards |
| MCP-Universe Agents complete tasks through 11 real MCP servers across six domains, such as navigation, repositories, finance, and browsing. Scored by execution-based success rate. | University | Aug 2025 | | Fading |
| MCPTox Agents use tools whose descriptions contain untrusted instructions. The benchmark measures whether agents follow those instructions across 1,348 cases from 45 servers. | University | Aug 2025 | | Not in cards |
| MLE-bench 30 Agents train models for 30 of MLE-bench's Kaggle-style competitions. The score is the share of runs that earn a bronze medal or better. | Expert | Aug 2025 | 36.7% | System cards only |
| MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. | Expert | Aug 2025 | 24.3% | Fading |
| MMSearch-Plus Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key. | Expert | Aug 2025 | | Fading |
| OdysseyBench Agents recover requirements from conversation histories and complete workflows across documents, spreadsheets, PDFs, email, and calendars. Task-specific checks inspect the resulting file system and report the percentage of completed tasks. | University | Aug 2025 | 58.7% | Not in cards |
| OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. | Research frontier | Aug 2025 | 8.00% | System cards only |
| Recovery-Bench Agents repair a task after a weaker agent leaves failed actions and a corrupted environment. The score is the share of recovery tasks that pass the original task verifier. | Expert | Aug 2025 | Not in cards | |
| ReportBench Agents write literature surveys. The evaluator compares their references with published surveys and checks whether cited sources support their claims. | Expert | Aug 2025 | 0.39 | Not in cards |
| Research-Eval Search-augmented models answer 374 grounded multi-hop questions that need sources across many kinds of websites. An LLM judge checks each answer against a checklist of requirements. | Expert | Aug 2025 | 66% | Dropped |
| ScrabbleBench Models choose a word from a Scrabble rack and board. A solver finds its best legal placement. The score averages the four seats in self-play. | Everyday | Aug 2025 | 152.8 | Not in cards |
| Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. | Expert | Aug 2025 | 99% | System cards only |
| TRAJECT-Bench Agents select and execute tools to answer queries. The evaluation checks whether the actions satisfy a reference trajectory and produce the correct answer. | University | Aug 2025 | | Not in cards |
| UQ Models answer 500 unsolved Stack Exchange questions. A model checks answers without a reference solution, and community members review selected answers. | Research frontier | Aug 2025 | Not in cards | |
| WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. | Everyday | Aug 2025 | | In use |
| Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. | Expert | Jul 2025 | 67% | System cards only |
| AlgoTune Agents rewrite code for 154 math, physics, and computer science functions to run faster and stay correct. Scoring is harmonic mean speedup over reference code. | Expert | Jul 2025 | 2.05× | Not in cards |
| Asper-Bench Agents turn complex user requests into programs that execute multi-step actions in a simulated digital assistant environment. Automated checks compare the resulting state with the requested goal. | Expert | Jul 2025 | Not in cards | |
| BFCL V4 Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. | Everyday | Jul 2025 | | In use |
| CC-Bench Claude Code runs realistic frontend, backend, tooling and data coding tasks in fresh containers. Humans compare models head-to-head (win rate). | Expert | Jul 2025 | 41.9% | System cards only |
| Emergent Price-Fixing by LLM Auction Agents Models trade in simulated auctions and communicate with other participants. A judge examines game logs for evidence of price fixing. | Everyday | Jul 2025 | 19% | Not in cards |
| FantasyFootballBench Models draft players, set weekly lineups, and negotiate trades in a replay of the 2024 NFL season. Actual player statistics determine points and wins. | Everyday | Jul 2025 | 15.0 | Not in cards |
| Galileo Agent Leaderboard v2 Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality. | Everyday | Jul 2025 | 62% | Not in cards |
| Gray Swan Agent Red Teaming (ART) benchmark Agents face curated prompt injections from a large public red-teaming competition across 44 deployment scenarios. Score is attack success rate. | Expert | Jul 2025 | 0.10% | System cards only |
| Group Binary Search (Riedl) Agents each choose an integer without seeing the other guesses. They adjust after hearing whether the group sum exceeds or falls below a hidden target. | Everyday | Jul 2025 | | Not in cards |
| Husky Hold'em Bench Models write Python poker bots and revise them after practice games. Six bots compete at each table, and the tournament tracks average winnings. | University | Jul 2025 | Not in cards | |
| In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. | Expert | Jul 2025 | 78% | System cards only |
| Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. | Expert | Jul 2025 | 88.6% | Fading |
| Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. | Expert | Jul 2025 | 99.5% | System cards only |
| Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. | Expert | Jul 2025 | 95% | System cards only |
| Labelbox Deep Research Agents Agents investigate research questions at the doctoral level. Labelbox compares their answers and reports scores by domain. | Expert | Jul 2025 | | Not in cards |
| LLM Economist Models set tax rates or choose working hours in a simulated economy. Researchers compare the social welfare achieved under different tax policies. | University | Jul 2025 | Not in cards | |
| MMBench-GUI GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration. | Everyday | Jul 2025 | | System cards only |
| Multi-SWE-bench flash Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing. | Expert | Jul 2025 | | Dropped |
| OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. | Everyday | Jul 2025 | 53.4% | Not in cards |
| OSWorld-Verified 369 open-ended tasks in real desktop apps. Agent acts through screenshots, mouse and keyboard. Execution-based checkers score success. | Everyday | Jul 2025 | 86% | In use |
| PACT Two models bargain over 20 rounds as buyer and seller, each with a hidden private value. Matches across many pairings rank the models. | Everyday | Jul 2025 | 1,607 | Not in cards |
| RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. | Everyday | Jul 2025 | | Fading |
| SLDBench Agents infer scaling laws from data covering more than 5,000 language model experiments. Eight tasks test how well their formulas predict results at other scales. | Expert | Jul 2025 | Not in cards | |
| SnorkelWordle Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate. | Everyday | Jul 2025 | 94% | Not in cards |
| SWE-Perf Agents speed up code in 140 tasks drawn from performance pull requests in real GitHub repos. Scored on correctness and runtime gain. | Expert | Jul 2025 | 2.26% | Fading |
| TextQuests Agents play 25 classic interactive fiction games. The score is harmful actions per 100 steps, where lower is better. The board also reports game progress. | Everyday | Jul 2025 | | System cards only |
| UserBench Agents help simulated users who start with vague goals and reveal preferences bit by bit. Agents must ask clarifying questions and use tools to reach choices that fit the user. | Everyday | Jul 2025 | 0.61 | Not in cards |
| WISER-Search Web research agents answer fresh and business questions that require current sources, multi-hop search, and evidence synthesis. | Expert | Jul 2025 | Not in cards | |
| Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. | Expert | Jun 2025 | 0% | In use |
| AIRTBench Models act as autonomous AI red teamers on 70 black-box capture-the-flag challenges from Dreadnode's Crucible platform. Each challenge hides a flag behind an AI or ML system, and the model writes Python code to attack it with methods such as prompt injection, evasion and model inversion. A challenge counts as solved when the model submits the correct flag. | Expert | Jun 2025 | 46.9% | Not in cards |
| ALE-Bench Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants. | Competition | Jun 2025 | 3,164 | Not in cards |
| AssetOpsBench Agents automate industrial asset work, such as condition monitoring, maintenance planning and intervention scheduling, using sensor data, failure models and work orders. | Expert | Jun 2025 | 65% | Not in cards |
| Automated LLM Speedrunning Benchmark Agents try to reproduce 19 past records of the NanoGPT speedrun, given the earlier code and hints of varying detail. Scored by how much of each record's speedup they recover. | Research frontier | Jun 2025 | 46% | Not in cards |
| BashBench Agents complete multi-step system administration tasks while red teams attempt covert malicious actions in the same shell environment. | Expert | Jun 2025 | Not in cards | |
| Bench to the Future Agents forecast hundreds of already-resolved questions while researching an offline corpus of web pages saved before the resolution. Scored by Brier score. | Expert | Jun 2025 | Not in cards | |
| BIRD-Interact Models solve ambiguous database tasks by talking with a simulated user, in a conversational mode and an agentic mode. 600 BI and CRUD tasks are checked by executable tests. | Expert | Jun 2025 | 25.5% | Not in cards |
| CVDP Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs. | Expert | Jun 2025 | 64.1% | Not in cards |
| CyberGym Large-scale benchmark of real open-source vulnerabilities. Agent must reproduce each known vulnerability with a proof-of-concept input. | Expert | Jun 2025 | 79.7% | In use |
| DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. | Expert | Jun 2025 | Not in cards | |
| DeepResearch Bench Agents write cited research reports for 100 PhD-level tasks across 22 fields. Judges score report quality against references and check citation accuracy. | Expert | Jun 2025 | 55.3% | Fading |
| DeepShop Web agents answer complex shopping queries on live sites with product attributes, search filters and sorting preferences. An LLM judge checks whether results meet each requirement. | Everyday | Jun 2025 | | Fading |
| FieldWorkArena V2.0 Agents inspect workplace images, videos, manuals, and logs to plan actions and report operational events. The evaluator compares their answers and actions with reference results. It combines correctness judgments with partial credit for numerical agreement. | University | Jun 2025 | 52% | Not in cards |
| Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. | Expert | Jun 2025 | 53.1% | Not in cards |
| GeoBenchX v2 Agents use GIS tools to join data, measure spatial relationships, and draw maps. Three model judges compare their work with corrected reference solutions. | University | Jun 2025 | 84% | Not in cards |
| Hangman Bench Agents guess letters to identify hidden words in Hangman. The harness tracks wins across difficulty levels. | Everyday | Jun 2025 | Not in cards | |
| HexMachina Models write Catan players in code and compete in the board game. | University | Jun 2025 | Not in cards | |
| Labelbox Agentic Search Agents search for answers to 200 questions across several knowledge domains. Labelbox reports overall and domain scores. | University | Jun 2025 | | Not in cards |
| MCP-SIM Agents turn physics problems into FEniCS simulations. The tasks include heat flow, elastic deformation, and fracture. The authors count solved tasks. | University | Jun 2025 | Not in cards | |
| Mind2Web 2 Agents browse the live web for 130 long tasks and return answers with citations. Task-specific judge agents built from rubric trees grade correctness and attribution. | Expert | Jun 2025 | 28% | Not in cards |
| OPT-BENCH Agents iteratively improve solutions to 20 machine-learning tasks and 10 combinatorial optimization problems using execution feedback. Win counts, invalid-solution rates, ranks, and improvement rates measure optimization performance. | University | Jun 2025 | | Not in cards |
| Orak LLM agents play 12 popular video games across most genres through an MCP interface. Each game scores in-game progress or outcomes, and the board ranks models by their average rank across the games. | Everyday | Jun 2025 | | Fading |
| RExBench Agents extend 12 published AI research projects from expert instructions. The evaluator runs their code against task-specific success criteria. Harbor distributes two public tasks. | Research frontier | Jun 2025 | 50% | Not in cards |
| RoboArena Double-blind pairwise A/B trials of generalist DROID policies on real Franka arms, free-form tasks by volunteer evaluators. Elo-style rating. | Everyday | Jun 2025 | 1,608 | Not in cards |
| RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. | Everyday | Jun 2025 | 58.3% | Not in cards |
| ScreenSuite Models use screenshots to locate controls, select actions and complete computer tasks. The suite measures perception, single actions and complete interactions. | Everyday | Jun 2025 | | Not in cards |
| Seal-0 Models answer 111 fact-seeking questions where web search returns conflicting or unhelpful results. Answers are graded for accuracy, usually with search enabled. | University | Jun 2025 | 57.4% | Fading |
| SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. | Expert | Jun 2025 | 18% | Not in cards |
| SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. | Jun 2025 | 26.2% | System cards only | |
| tau2-bench Dual-control customer-service simulations (retail, airline, telecom) where agent and simulated user both act. Graded on final state. | Everyday | Jun 2025 | 98.9% | In use |
| ts-bench Coding agents such as Claude Code, Codex and Gemini CLI solve TypeScript tasks, and tests decide pass or fail. Version 1 uses 25 Exercism practice exercises. Version 2 uses real tasks from a large TypeScript monorepo, taken from SWE-Lancer. | University | Jun 2025 | 80% | Not in cards |
| ts-bench v1 Coding agents solve 25 self-contained TypeScript exercises from Exercism. The score is the percentage of exercises that pass their tests. | University | Jun 2025 | 96% | Not in cards |
| V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. | Everyday | Jun 2025 | Not in cards | |
| Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. | Everyday | Jun 2025 | 84.9% | Not in cards |
| WorldPrediction Agents predict and plan in procedural environments with long horizons. Tasks test whether models maintain a useful world model while acting. | University | Jun 2025 | Not in cards | |
| xbench-Marketing Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers. | Expert | Jun 2025 | 50.8 | Not in cards |
| xbench-Recruitment Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5. | Expert | Jun 2025 | 78.5 | Not in cards |
| ADE-bench Agents repair dbt projects so that SQL tests pass on the resulting tables. Harbor ports 48 DuckDB tasks and excludes the Snowflake and dbt Fusion variants. | Expert | May 2025 | Not in cards | |
| Agentic coding malicious requests (Anthropic) Coding agents with standard coding tools receive 150 malicious coding requests prohibited by policy. Score is the refusal rate. | Expert | May 2025 | 100% | System cards only |
| AndroidWorld-Knowledge Agents attempt 48 AndroidWorld tasks across five apps. Each app provides guidance for its simplest task, and harder tasks test whether the agent can reuse it. | Everyday | May 2025 | Not in cards | |
| BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. | Expert | May 2025 | 50% | Not in cards |
| C3-Bench Agents complete multi-task tool use dialogues built to stress tool dependencies, long context dependencies and frequent switches between policy types. | University | May 2025 | 12.5% | Dropped |
| Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. | Expert | May 2025 | 100% | System cards only |
| Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy. | Expert | May 2025 | Dropped | |
| CRMArena-Pro Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private. | Everyday | May 2025 | | Not in cards |
| Deep Research Bench (FutureSearch) Web research agents work on multi-step research tasks against RetroSearch, a frozen copy of 10,000 to 100,000 web pages per task. Answers are compared with answers worked out by people. | Expert | May 2025 | 0.55 | Not in cards |
| DeepConsult Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer. | Expert | May 2025 | | Fading |
| DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. | Expert | May 2025 | 80% | System cards only |
| End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score. | Expert | May 2025 | Dropped | |
| EXP-Bench Agents get a research question and incomplete starter code from an influential AI paper. They must form hypotheses, design and run experiments, and analyze results. | Research frontier | May 2025 | 0.50% | Not in cards |
| FoamBench Agents create OpenFOAM cases from written specifications. The grader checks whether the solver finishes and whether the final flow field matches a reference simulation. | University | May 2025 | 22.5% | Not in cards |
| GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. | Expert | May 2025 | | System cards only |
| GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. | Expert | May 2025 | | System cards only |
| gg-bench Models play 126 games invented by a language model and receive game-specific outcomes. | Everyday | May 2025 | 36.3% | Not in cards |
| GSO-Bench Agents speed up code in 102 tasks across 10 real codebases, given a performance test. Scored by the share matching the expert commit's speedup. | Expert | May 2025 | 88.2% | Not in cards |
| Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. | Expert | May 2025 | 30% | System cards only |
| Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. | Expert | May 2025 | 0.65 | System cards only |
| Kernels task (Anthropic) Agents optimize compute kernels in a performance engineering challenge. Scored by speedup. | Expert | May 2025 | System cards only | |
| KORGym Models play 51 text and visual games designed to test interactive reasoning. | Everyday | May 2025 | 0.82 | Not in cards |
| LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. | Everyday | May 2025 | 72.4% | Fading |
| Letta Leaderboard Agents read, write and update core and archival memory while answering synthetic factual questions. A model judge scores answers and penalizes unnecessary memory operations. | Expert | May 2025 | 93.0 | Not in cards |
| lmgame-Bench Models play platformer, puzzle and narrative games such as Sokoban, Tetris, 2048 and Candy Crush through a Gym-style API with perception and memory scaffolds. | Everyday | May 2025 | 87.7 | Not in cards |
| Novel compiler task (Anthropic) Agents build a compiler for an unusual programming language from a specification and test cases. | Expert | May 2025 | 80.9% | System cards only |
| ResearchCodeBench Agents read machine learning papers and fill missing Python code in the accompanying implementations. Unit tests check each of 212 snippets from 20 papers. | Expert | May 2025 | | Not in cards |
| Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. | Expert | May 2025 | 0% | System cards only |
| RiOSWorld Computer-use agents work in an Ubuntu virtual machine through risky scenarios such as phishing emails and pop-ups. Scores track whether agents notice and avoid the risk. | Everyday | May 2025 | | Not in cards |
| ScienceBoard Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking. | Expert | May 2025 | 68% | Not in cards |
| Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. | Expert | May 2025 | System cards only | |
| SnorkelUnderwrite Agents help simulated small business insurance underwriters across six task types in multi-turn chats, using tools and asking the underwriter for missing facts. | Expert | May 2025 | 91% | Not in cards |
| Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). | May 2025 | | System cards only | |
| SWE-bench-Live Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch. | Expert | May 2025 | | Dropped |
| SWE-bench-Live Verified Task success on the official Verified split. The source reports each submission denominator separately. Other language and platform splits remain separate. | Expert | May 2025 | 40% | Not in cards |
| SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail. | Expert | May 2025 | 66.6% | Dropped |
| SWE-rebench Agents fix fresh GitHub issues that an automated pipeline mines each month. Tests decide whether the issue is resolved. | Expert | May 2025 | 64.5% | System cards only |
| Terminal-Bench Agents complete about 80 hard tasks in sandboxed Linux terminals, such as compiling code or configuring servers. Test scripts check the final container state. | Expert | May 2025 | 49.6% | Fading |
| Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. | Expert | May 2025 | 0.97 | System cards only |
| ThinkGeo Agents use remote sensing tools to interpret aerial and satellite images. The authors compare individual tool calls and use GPT-4o mini to grade final answers. | University | May 2025 | 100% | Not in cards |
| Time series forecasting task (Anthropic) Agents solve a regression and time-series forecasting problem with known state-of-the-art results, in six variants. | Expert | May 2025 | 4.55 | System cards only |
| ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. | Everyday | May 2025 | | System cards only |
| Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. | Expert | May 2025 | 0.63% | System cards only |
| Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. | Expert | May 2025 | 85% | System cards only |
| VideoGameBench Vision-language models play 10 popular 1990s Game Boy and MS-DOS games in real time from raw frames. Scored by the share of each game completed. | Expert | May 2025 | | Not in cards |
| VideoGameBench Lite Agents play video games while the environment pauses during inference. The headline is the official Overall Score. | Expert | May 2025 | 1.60% | Not in cards |
| WebBench (Halluminate) Browser agents complete 2,454 tasks on 452 live websites from the global top 1,000 by traffic. Tasks either read information or write data: creating, updating and deleting records, logging in and downloading files. The board compares browser agent products. | Everyday | May 2025 | 76.5% | Not in cards |
| xbench-DeepSearch Agents search the web to answer 100 Chinese questions that need multi-step retrieval. Scored by accuracy against reference answers. | Expert | May 2025 | | Fading |
| AI Engineering Lab simulation self-validation Agents predict how a mechanical system should behave, inspect simulation outputs, and decide whether the model is correct. The authors score these decisions with F1. | University | Apr 2025 | 0.72 | Not in cards |
| Among Us: A Sandbox for Measuring and Detecting Agentic Deception Language agents play a multiplayer social-deception game with hidden roles. Deception Elo and detection Elo measure success over repeated matches. | Expert | Apr 2025 | Not in cards | |
| Apollo Research sabotage evaluations Agents are told to sabotage another model through backdoors, sandbagging finetunes or hidden communication. Scored as mean task score from 0 to 1. | Expert | Apr 2025 | 0.17 | System cards only |
| AutoCFD Incompressible Flow Benchmark Agents configure OpenFOAM simulations from flow descriptions and meshes. The authors check solver convergence and compare simulated flow with reference values. | University | Apr 2025 | | Not in cards |
| AutoPatchBench Agents write security patches for crashes that a fuzzer found in native code. Fixes are checked by fuzzing and differential debugging. | Expert | Apr 2025 | | Not in cards |
| BrowseComp Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. | Expert | Apr 2025 | 93.3% | In use |
| BrowseComp-ZH Agents browse the Chinese web to answer 289 multi-hop questions with short, verifiable answers. Scored by accuracy. | Expert | Apr 2025 | | Fading |
| CRUST-bench Agents translate 100 C projects into safe Rust. Tests check whether each translation matches the supplied Rust interface and preserves the original behavior. | Expert | Apr 2025 | 48% | Not in cards |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | Expert | Apr 2025 | 93.3% | System cards only |
| DrafterBench Agents call drawing revision tools from written instructions. The evaluator records their operations and arguments across 1,920 tasks covering 12 types of revision. | University | Apr 2025 | 84.5% | Not in cards |
| KUMO Agents make queries and use the responses to identify hidden facts in generated reasoning games. Harbor offers 5,050 easy games and 250 hard games. | University | Apr 2025 | | Not in cards |
| LLM-SRBench Models infer scientific equations from observations on 239 problems across four scientific domains. Symbolic accuracy checks whether the discovered equation recovers the underlying relationship. | Research frontier | Apr 2025 | | Not in cards |
| LMArena Search Arena People compare answers from two anonymous search-augmented models on their own questions and vote for the better one. Votes become Bradley-Terry ratings. | Everyday | Apr 2025 | 1,255 | Fading |
| m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. | Everyday | Apr 2025 | | Dropped |
| MafiaBench Models play a social deduction game with hidden roles and public discussion. | Everyday | Apr 2025 | | Not in cards |
| MLRC-Bench Agents propose and implement methods for machine learning competitions. The scorer compares their performance with a supplied baseline and the strongest human solution. | Research frontier | Apr 2025 | | Not in cards |
| Multi-SWE-bench Agents resolve 1,632 GitHub issues in Java, TypeScript, JavaScript, Go, Rust, C, and C++ repos. Patches must pass the repos' tests. | Expert | Apr 2025 | | Fading |
| Multi-SWE-bench mini Agents fix real GitHub issues in 8 programming languages, a 400-instance subset of Multi-SWE-bench made for cheaper runs. Scored by tests passing. | Expert | Apr 2025 | | Dropped |
| Online-Mind2Web Web agents do 300 tasks on 136 live websites. An LLM judge reads the agent's trajectory and decides whether the task succeeded. | Everyday | Apr 2025 | | Fading |
| Open RAG Benchmark RAG systems answer 3,045 queries over 1,000 arXiv PDFs with text, tables, images, and hard negatives. | Expert | Apr 2025 | Not in cards | |
| PaperBench Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. | Research frontier | Apr 2025 | | In use |
| PaperBench Code-Dev Agents implement research paper code. The Code-Dev variant evaluates coding separately from full research replication. | Research frontier | Apr 2025 | 43.4% | Not in cards |
| Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. | Expert | Apr 2025 | System cards only | |
| REAL Agents complete 112 tasks on replicas of 11 websites. State checks grade actions, while a model judge checks retrieved answers against references. | Everyday | Apr 2025 | 41.1% | Not in cards |
| RepliBench Agents acquire resources, exfiltrate model weights, replicate onto compute and persist over time. Task checks measure components of autonomous self-replication. | Research frontier | Apr 2025 | Not in cards | |
| SWE-bench Multilingual 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. | Expert | Apr 2025 | 96.6% | In use |
| SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution. | Expert | Apr 2025 | 14.1% | Not in cards |
| SWE-smith Agents repair software faults generated from real repositories. Harbor evaluates a 100-task sample from the larger training collection using tests for each fault. | Expert | Apr 2025 | 38% | Not in cards |
| TextArena Collection of competitive text games for LLM eval and RL, with an online Elo leaderboard. | Apr 2025 | Not in cards | ||
| Vals Finance Agent v1 Agents answer 537 financial analyst questions on public company SEC filings using search and filing tools. An LLM judge grades each final answer. | Expert | Apr 2025 | 60.6% | Dropped |
| WISER-Atomic Web research systems answer 121 customer-inspired questions across finance, sales, recruitment, and other domains. | Expert | Apr 2025 | 69% | Not in cards |
| WixQA Models retrieve Wix support articles and answer customer questions from those articles. The authors score retrieval and answer generation separately. | Everyday | Apr 2025 | Not in cards | |
| AI Agent Privacy Web agents complete tasks while exposed to private information. The benchmark measures privacy leakage and unsafe disclosure during autonomous operation. | Expert | Mar 2025 | Not in cards | |
| AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. | Expert | Mar 2025 | 22% | Not in cards |
| AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. | Everyday | Mar 2025 | | Not in cards |
| BLUR Models identify a remembered item from incomplete clues in several languages and media. | Everyday | Mar 2025 | 54% | Not in cards |
| CVE-Bench Real-world web-application CVEs in sandboxed services. Agent is scored on reaching predefined attack objectives. | Expert | Mar 2025 | 94.1% | System cards only |
| Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. | Expert | Mar 2025 | | System cards only |
| Dubesor AI Chess Leaderboard Models play chess by continuing game notation or by reasoning from the board and legal moves. The organizer maintains separate Elo ratings for each mode. | School | Mar 2025 | | Not in cards |
| Factorio Learning Environment Agents write Python programs that control a Factorio factory, in lab tasks with fixed resources and in open play. Scored by production throughput, milestones and lab task success. | Expert | Mar 2025 | 293206.0 | Not in cards |
| Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. | Everyday | Mar 2025 | Not in cards | |
| GeoBenchX Agents use GIS tools to solve spatial analysis tasks or identify missing data. Three model judges compare the proposed workflows with reference solutions. | University | Mar 2025 | 51.5% | Not in cards |
| HCAST Agents attempt 189 machine learning, cybersecurity, software engineering and reasoning tasks. 563 human baselines time each task, from one minute to over 8 hours, so success can be read against human task length. | Expert | Mar 2025 | Not in cards | |
| IVISTA Voice Interaction, revised 2023 protocol In-car assistants carry out spoken commands from different seats and answer follow-up requests. Testers check completed actions, including window adjustments and navigation commands. | Everyday | Mar 2025 | Not in cards | |
| METR Time Horizon Agents attempt software and research tasks timed against skilled humans. The score is the human task length at which the agent succeeds half the time. | Research frontier | Mar 2025 | 4.1 h | System cards only |
| METR Time Horizon suite Software tasks with human completion times. Reports the task length at which a model succeeds 50% of the time. | Research frontier | Mar 2025 | 4.1 h | Not in cards |
| RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. | Everyday | Mar 2025 | | Fading |
| TREC Product Search 2025 Models rewrite shopping queries for a search engine. Automatic entries submit one reformulation. Interactive entries can submit four, and the best one counts. | Everyday | Mar 2025 | Not in cards | |
| AI Diplomacy Models play the board game Diplomacy against each other, one power each. They negotiate, form alliances and betray each other in written messages. | Everyday | Feb 2025 | | Not in cards |
| BenchMAX Models follow instructions, write code, reason, use tools, translate, and retrieve information across 17 languages. Ten tasks use rule checks, model-judged wins, code tests, answer matching, or translation scores. | University | Feb 2025 | | Not in cards |
| BixBench Computational-biology analysis tasks (differential expression, assembly QC, cell typing) answered by agents with tools. | Expert | Feb 2025 | 93.8% | Fading |
| Collab-Overcooked Two agents exchange messages and use different kitchen equipment to fill orders. The benchmark measures task completion and progress through required cooking steps. | Everyday | Feb 2025 | 86.7% | Not in cards |
| ControlArena Agents try to perform unwanted side actions stealthily while completing a legitimate user task. Scored on whether monitors catch them. | Expert | Feb 2025 | System cards only | |
| Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. | Expert | Feb 2025 | 94.4% | System cards only |
| Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. | Research frontier | Feb 2025 | 66.7% | System cards only |
| Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. | Expert | Feb 2025 | System cards only | |
| Elimination Game Models play a multi-player game of public and private talk, alliances and votes to eliminate each other. A jury of eliminated players picks the winner. | Everyday | Feb 2025 | 8 | Not in cards |
| EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. | Everyday | Feb 2025 | 50.5% | Not in cards |
| EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. | Everyday | Feb 2025 | 67.7% | Not in cards |
| EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. | Everyday | Feb 2025 | 58.7% | Not in cards |
| EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. | Everyday | Feb 2025 | 28.5% | Not in cards |
| EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. | Everyday | Feb 2025 | | Not in cards |
| Galileo Agent Leaderboard Models pick and call tools on samples drawn from existing tool-use benchmarks. Galileo scores Tool Selection Quality with an LLM judge and averages the datasets equally. | Everyday | Feb 2025 | 0.95 | Not in cards |
| ITBench Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. | Expert | Feb 2025 | | Not in cards |
| KernelBenchFiltered Agents optimize GPU kernels. METR takes the geometric mean of the speedups across tasks after checking correctness. | Expert | Feb 2025 | Not in cards | |
| LLM Public Goods Game Agents choose how many tokens to contribute to a common pool. The evaluation reports the fraction each model contributes across repeated games. | Everyday | Feb 2025 | | Not in cards |
| LLM training task (Anthropic) Agents optimize a CPU-only small language model training implementation. Scored by speedup. | Expert | Feb 2025 | System cards only | |
| Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. | Expert | Feb 2025 | 69.7% | System cards only |
| Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. | Everyday | Feb 2025 | 99.4% | System cards only |
| MLGym-Bench Agents work on 13 open-ended AI research tasks in vision, language, reinforcement learning and game theory. Scores compare the agent's result with baselines on each task's metric. | Expert | Feb 2025 | | Not in cards |
| Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. | Everyday | Feb 2025 | System cards only | |
| Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. | Expert | Feb 2025 | 71.4% | System cards only |
| Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. | Expert | Feb 2025 | 31.3 | System cards only |
| RE-Bench subset (Anthropic) Agents solve a modified subset of four METR RE-Bench research engineering tasks. Scores are normalized from reference solution 0 to expert solution 1. | Expert | Feb 2025 | 0.24 | System cards only |
| Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| SnakeBench Two language models each steer a snake on a shared board in a turn-based game, and the one that survives longest wins. Ratings come from many head-to-head games. | Everyday | Feb 2025 | 39 | Not in cards |
| SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. | Expert | Feb 2025 | 48.2% | System cards only |
| SWE-Lancer Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned. | Expert | Feb 2025 | 81.4% | Fading |
| Vals Legal AI Report Legal AI tools perform seven kinds of legal work. Vals compares their task scores with scores from practicing lawyers. | Expert | Feb 2025 | Not in cards | |
| Vending-Bench Agents run a simulated vending machine business over long horizons, ordering stock, setting prices, and paying daily fees. Scoring is final net worth in dollars. | Everyday | Feb 2025 | $4,694 | Dropped |
| Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| WorldGUI Desktop GUI agents do 611 tasks that start from varied initial states, such as the target app not being open. Tasks come with instructional videos and project files. | Everyday | Feb 2025 | 45.8% | Not in cards |
| Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. | Everyday | Jan 2025 | 97% | System cards only |
| Android Agent Arena (201 tasks) Agents find live information and perform actions in Android apps such as YouTube Music and Booking.com. Task functions check app state and grade answers. | Everyday | Jan 2025 | | Not in cards |
| Autonomy-10 Agents complete browser, desktop and tool-use tasks across nine domains that measure skill, flexibility and intelligence. | Expert | Jan 2025 | Not in cards | |
| ComplexFuncBench Models make function calls for 1,000 travel booking style samples that need several steps, user constraints, implicit parameter reasoning, long parameter values or 128K context. | University | Jan 2025 | 61% | Dropped |
| CSR-Bench Agents deploy and run computer-science research repositories with shell, retrieval and web-search tools. Task success measures whether the requested repository workflow completes. | Expert | Jan 2025 | Not in cards | |
| gBench Models run academic evaluations, Golden Set smoke tests, and an agentic quality suite through an OpenAI-compatible endpoint. The tool also records serving performance. | University | Jan 2025 | Not in cards | |
| MedAgentBench Agents use FHIR APIs in a virtual electronic health record to do 300 patient tasks in 10 categories written by physicians. | Expert | Jan 2025 | 69.7% | Not in cards |
| ML-Dev-Bench Agents implement and train models, debug failures, and improve performance. Harbor ports 33 development tasks and checks them with the original validators. | Expert | Jan 2025 | Not in cards | |
| Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. | Everyday | Jan 2025 | Not in cards | |
| OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. | Research frontier | Jan 2025 | 55% | System cards only |
| Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. | Expert | Jan 2025 | 23% | System cards only |
| Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. | Everyday | Jan 2025 | 94% | System cards only |
| Step Race Several models talk in public, then secretly pick moves in a race where colliding moves cancel. The game tests cooperation and deception. | Everyday | Jan 2025 | 5 | Not in cards |
| ToolComp Models chain several tool calls to answer 485 human-verified prompts. Most final answers are numbers. ToolComp-Enterprise gives 11 tools and ToolComp-Chat targets general chat tool use. Step labels also support process reward model tests. | Expert | Jan 2025 | 70.1% | Not in cards |
| User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. | Everyday | Jan 2025 | 92% | System cards only |
| WebWalkerQA Agents start from a website's home page and traverse its subpages to find the facts that answer a question. Answers are judged against references. | Everyday | Jan 2025 | | Not in cards |
| SnorkelFinance ReAct agents answer expert-verified financial questions built from 10-K filings, using SQL, code and other MCP tools without seeing the documents. A judge scores full traces. | University | 2025 | 81% | Not in cards |
| Agent-SafetyBench LLM agents act in 349 tool environments over 2,000 test cases covering 8 safety risk categories and 10 failure modes. A scorer model rates each run safe or unsafe. | Everyday | Dec 2024 | 59.8% | Not in cards |
| Aider Polyglot Models edit code to solve 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust. Scored by the share passing unit tests. | University | Dec 2024 | 88% | Fading |
| Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. | Expert | Dec 2024 | System cards only | |
| Domain Intelligence Benchmark Suite Models extract structured data, call functions, and answer questions using retrieved material. Databricks compares performance on enterprise tasks. | University | Dec 2024 | Not in cards | |
| HackSynth OverTheWire Agents complete scored security exercises from OverTheWire. The published benchmark covers 80 challenges, and the hub port selects 46. | Expert | Dec 2024 | | Not in cards |
| HackSynth PicoCTF Models attempt 120 tasks selected from picoCTF. The benchmark measures the percentage of tasks completed, and the hub port checks submitted answers against reference flags. | Competition | Dec 2024 | | Not in cards |
| LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. | University | Dec 2024 | Not in cards | |
| RelyToolBench Agents answer queries with simulated APIs built on the StableToolBench server. Scoring targets tool hallucinations, such as calling wrong or missing tools. | Everyday | Dec 2024 | | In use |
| Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. | Expert | Dec 2024 | System cards only | |
| SuperCLUE-Hard Models follow precise instructions, solve reasoning problems and plan tool calls in Chinese. The report gives separate scores for each ability. | University | Dec 2024 | 76.7 | Not in cards |
| TheAgentCompany Agents do work tasks in a simulated software company, using a browser, code, and chat with simulated coworkers. Scored by full and partial checkpoint completion. | University | Dec 2024 | Dropped | |
| VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. | Everyday | Dec 2024 | | Not in cards |
| BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. | Everyday | Nov 2024 | 68.3% | Not in cards |
| CRMArena Agents do nine customer-service CRM tasks for service agent, analyst and manager personas inside a real Salesforce org. The org holds 16 linked object types with seeded hidden patterns. | Everyday | Nov 2024 | 64.3% | Not in cards |
| PARTNR Agents follow natural language instructions in human-robot collaboration scenes. Tasks require planning and coordination among multiple embodied agents. | University | Nov 2024 | Not in cards | |
| RedCode Models meet adversarial prompts to generate or run harmful code. Generation is model-judged and execution is checked in a container. | Expert | Nov 2024 | | System cards only |
| Spider 2.0 Models and agents write SQL for real enterprise databases with large schemas, across BigQuery, Snowflake, SQLite and dbt projects. Answers are checked by executing them. | Expert | Nov 2024 | | Dropped |
| Agent Arena (Gorilla) Users give a task, two agents built from different models, tools and frameworks run it, and users vote. A Bradley-Terry extension rates models, tools and frameworks. | Everyday | Oct 2024 | | Not in cards |
| AgentHarm 110 malicious agent tasks with synthetic tools. Measures refusal and harmful task completion, plus benign controls. | Oct 2024 | 8.00% | Fading | |
| AndroidLab Mobile agents complete 138 tasks across nine apps on predefined Android virtual devices, in text-only and multimodal modes. Checks on the final device state score each task. | University | Oct 2024 | 31.2% | Not in cards |
| AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. | Expert | Oct 2024 | Not in cards | |
| BrowserART Browser agents receive 100 harmful requests on synthetic and real websites. The test measures how often the agent attempts the harmful behavior that its backbone model would refuse in chat. | Expert | Oct 2024 | 11% | Not in cards |
| Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. | Expert | Oct 2024 | | Not in cards |
| Embodied Agent Interface Models interpret goals, decompose them into subgoals, sequence actions, and predict state transitions in BEHAVIOR and VirtualHome. The evaluator checks task completion and distinguishes hallucinations, invalid actions, and planning errors. | Everyday | Oct 2024 | 70.3 | Not in cards |
| GLEE Agents bargain, negotiate, or persuade another player through conversation while pursuing the payoff defined by the game. | University | Oct 2024 | 2,595 | Not in cards |
| ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. | Everyday | Oct 2024 | Not in cards | |
| MLE-bench Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. | Expert | Oct 2024 | 96.1% | In use |
| ScienceAgentBench Agents write programs for 102 data-driven discovery tasks from 44 papers in four disciplines. Outputs are checked by execution and task-specific metrics. | Expert | Oct 2024 | 42.2% | Not in cards |
| SPA-Bench Agents use Android apps such as Airbnb, Spotify, and WeChat. The grader matches screenshots against task requirements and uses a vision model to resolve uncertain cases. | Everyday | Oct 2024 | 1.35% | Not in cards |
| SWE-bench Multimodal SWE-bench tasks whose issues include screenshots or design mockups from visual JavaScript repos. Patch graded by hidden tests. | Expert | Oct 2024 | 74.1% | In use |
| Verus-Bench Agents synthesize Verus proof annotations for 150 Rust verification tasks from algorithms, arrays and loop programs. | Expert | Oct 2024 | Not in cards | |
| WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. | Oct 2024 | Not in cards | ||
| Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. | Expert | Sep 2024 | System cards only | |
| cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability. | Expert | Sep 2024 | 51.8% | Not in cards |
| ContextualBench CRM agents answer enterprise questions against contextual business data and receive accuracy, cost, speed, trust and safety measurements. | Expert | Sep 2024 | Not in cards | |
| CORE-Bench Agents reproduce the results of published studies from their code and data. 270 tasks from 90 papers in computer science, social science and medicine come in three difficulty levels. | Expert | Sep 2024 | 77.8% | Not in cards |
| DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. | University | Sep 2024 | Not in cards | |
| DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap. | Expert | Sep 2024 | 89.9% | Dropped |
| FlagEval Debate Models debate a topic in Chinese. The leaderboard estimates their ratings from comparisons and reports uncertainty. | Everyday | Sep 2024 | 1,149 | Not in cards |
| ForecastBench Models give probabilities for 500 auto-generated questions per round about future events, drawn from prediction markets and time-series datasets. Forecasts are scored with difficulty-adjusted Brier scores once the questions resolve. | Expert | Sep 2024 | 0.11 | In use |
| FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. | Everyday | Sep 2024 | 87.6% | Fading |
| LLM Chess Models play chess in multi-turn dialogs against a random player or the Komodo Dragon engine. The board reports wins, Elo and game duration, which tracks instruction following. | Everyday | Sep 2024 | 1,614 | Not in cards |
| RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. | Research frontier | Sep 2024 | | System cards only |
| RedTeam Arena Players get 60 seconds to make a model say a target word. An extended Elo ranks players, models and target prompts, with models ranked by resistance. | Everyday | Sep 2024 | Not in cards | |
| RoboTwin 1.0 Dual-arm policies train on generated digital-twin demos and are tested on simulated single-arm and dual-arm tasks, with matching real-world data. Success rate. | Everyday | Sep 2024 | Not in cards | |
| Windows Agent Arena Agents complete more than 150 tasks in a real Windows OS with its apps, tools and browsers. Scripts adapted from OSWorld check the final state. | Everyday | Sep 2024 | | Dropped |
| Agentic Self-Reasoning (Apollo Research) Agents reason about their own operating conditions to complete tasks. Apollo measures success at three difficulty levels. | Everyday | Aug 2024 | Not in cards | |
| Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. | Expert | Aug 2024 | 78% | System cards only |
| Applied Theory of Mind (Apollo Research) Agents complete tasks that require tracking what another participant knows. Apollo measures success at three difficulty levels. | Everyday | Aug 2024 | Not in cards | |
| Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. | University | Aug 2024 | 68% | System cards only |
| Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. | School | Aug 2024 | 82% | System cards only |
| Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. | Expert | Aug 2024 | 96.3% | Fading |
| Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. | Expert | Aug 2024 | | Fading |
| HR-Bench Models answer multiple-choice questions about fine details in 4K and 8K resolution images. Results are reported separately for the 4K and 8K sets. | Everyday | Aug 2024 | | Fading |
| Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. | Expert | Aug 2024 | 0% | System cards only |
| SWE-bench Verified 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. | Expert | Aug 2024 | 83.5% | In use |
| Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. | Everyday | Aug 2024 | System cards only | |
| ToolSandbox Models use stateful tools in multi-turn conversations with a simulated user. Tools have hidden state dependencies, and runs are scored against milestones. | Everyday | Aug 2024 | 80.7% | In use |
| VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. | Everyday | Aug 2024 | 36.2% | Not in cards |
| Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. | Everyday | Aug 2024 | 5.74% | System cards only |
| XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. | Expert | Aug 2024 | 98.5% | Fading |
| AppWorld Agents write code against 457 APIs of 9 simulated everyday apps to finish 750 tasks for about 100 fictional users. State-based unit tests check the outcome and collateral damage. | Everyday | Jul 2024 | Not in cards | |
| AppWorld Test-Challenge Agents complete 417 app tasks that require Amazon or Gmail APIs held out from training. The score is the percentage passing every task test. | Everyday | Jul 2024 | 92.8% | Not in cards |
| AppWorld Test-Normal Agents complete 168 everyday app tasks from familiar applications by calling APIs through code. The score is the percentage passing every task test. | Everyday | Jul 2024 | 94% | Not in cards |
| AssistantBench Web agents do 214 realistic, time-consuming tasks on the open web, such as monitoring real-estate markets. Answers are checked automatically against gold answers. | Everyday | Jul 2024 | Not in cards | |
| ChiPBench (chip placement) Agents place circuit blocks in chip designs. OpenROAD measures timing, power, and area after routing. Higher timing slack is better. | Expert | Jul 2024 | Not in cards | |
| CIBench Agents use code interpreters to analyze data. The evaluation checks stages of the interactive analysis. | University | Jul 2024 | 75.5% | Not in cards |
| CRAB Multimodal agents do tasks that span an Ubuntu desktop and an Android phone at once. Graph-based evaluators give credit for intermediate steps. | Everyday | Jul 2024 | 38% | Not in cards |
| CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. | Expert | Jul 2024 | Not in cards | |
| CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. | Jul 2024 | | System cards only | |
| DiscoveryBench Agents analyze datasets and produce data-driven hypotheses that evaluators score against structured evidence and domain criteria. | Expert | Jul 2024 | Not in cards | |
| GTA Agents answer 229 human-written queries that need 1 to 4 real deployed tools for perception, operation, logic and creativity, often with image inputs. Queries leave the tools and steps implicit. | Everyday | Jul 2024 | 46.6% | Not in cards |
| LAB-Bench LitQA2 Models answer multiple-choice questions whose answers appear once in recent biology papers. Built to need literature search rather than recall. A LAB-Bench subtask. | Expert | Jul 2024 | 27% | Not in cards |
| RelBench Agents build models for prediction tasks over multi-table relational databases, graded on held-out targets. | Expert | Jul 2024 | 38.4% | System cards only |
| Spider2-V Multimodal agents do data science and engineering workflows in real enterprise apps such as BigQuery, dbt and Airbyte, on a desktop computer. Execution-based checks score each task. | Expert | Jul 2024 | 14% | Not in cards |
| Werewolf Arena Models compete in Werewolf with strategic speaking, deception, deduction, and persuasion. | Everyday | Jul 2024 | | Not in cards |
| WorkArena++ Web agents complete 682 compositional knowledge-work tasks in ServiceNow, which need planning, reasoning and memory across many steps. | Expert | Jul 2024 | 1.90% | Not in cards |
| AgentDojo Agents do tool-use tasks while tools and data carry prompt injections. Attack success rate counts hijacked runs. | Everyday | Jun 2024 | 0% | Fading |
| CharXiv Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. | University | Jun 2024 | 93.5% | In use |
| CRAG Retrieval-augmented systems answer 4,409 factual questions across finance, sports, music, movies and open domain, using mock web and knowledge graph search APIs. Hallucinated answers are penalized. | Everyday | Jun 2024 | 0.20 | Not in cards |
| Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass. | Expert | Jun 2024 | 78% | System cards only |
| LiveBench Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. | University | Jun 2024 | | Dropped |
| Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. | Everyday | Jun 2024 | Not in cards | |
| MobileAgentBench Agents complete 100 tasks across ten Android apps. The grader checks app state and accessibility events, so agents can take different paths. | Everyday | Jun 2024 | 8.00% | Not in cards |
| NanoGPT Speedrun Improve a small GPT training script to reach a target validation loss as fast as possible on fixed hardware. | Research frontier | Jun 2024 | 42.9% | System cards only |
| NATURAL PLAN Models make travel itineraries, find meeting arrangements, and schedule calendar events. The evaluator checks each proposed plan against the stated constraints. | Everyday | Jun 2024 | Not in cards | |
| NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. | University | Jun 2024 | 22% | Not in cards |
| RoboCasa Simulated mobile manipulators and humanoids do 100 kitchen tasks, 25 atomic and 75 composite, across 120 scenes. Scored by success rate. | Everyday | Jun 2024 | | Fading |
| SpreadsheetBench Real spreadsheet manipulation questions from Excel forums. Output workbook is checked cell-by-cell against reference solutions. | University | Jun 2024 | | In use |
| SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix. | Expert | Jun 2024 | | Fading |
| tau-bench Agents chat with a simulated user and call retail or airline APIs under policy rules. A task passes if the final database matches the goal. | Everyday | Jun 2024 | 70.5% | Dropped |
| tau-bench Airline Agents help simulated airline customers through tools while following airline policies on the original tasks. The score is pass^1 as a percentage. | Everyday | Jun 2024 | 46% | Not in cards |
| tau-bench Retail Agents help simulated retail customers through tools while following store policies on the original tasks. The score is pass^1 as a percentage. | Everyday | Jun 2024 | 67.8% | Dropped |
| WebCanvas Web agents do tasks on live websites, scored on key intermediate states rather than one fixed action path, so the test survives site changes. | Everyday | Jun 2024 | 28.8% | Not in cards |
| AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. | Expert | May 2024 | 37.8% | Not in cards |
| AnalogCoder Models write Python programs that build analog circuits. A simulator checks whether each circuit performs its requested function. The authors report first-attempt success and how many tasks each model solves. | University | May 2024 | | Not in cards |
| AndroidWorld Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. | Everyday | May 2024 | 100% | In use |
| MiniStS Models play card battles with five preset decks. The authors measure final health in four scenarios and how often agents play the Harm card in the fifth. | Everyday | May 2024 | Not in cards | |
| MobileMiniWoB++ Agents complete MiniWoB++ web tasks inside the Android emulator of AndroidWorld. Scored by task success rate. | Everyday | May 2024 | | Dropped |
| SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. | Everyday | May 2024 | 80% | In use |
| B-MoCA Agents use Android apps such as Instagram on phones with different languages, font sizes, and icon positions. Task-specific checks decide whether each attempt succeeded. | Everyday | Apr 2024 | 47.7% | Not in cards |
| CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. | University | Apr 2024 | | System cards only |
| CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. | Expert | Apr 2024 | Not in cards | |
| LlamaTouch Agents carry out requests in Android apps. The evaluator checks whether each run visited the required app or system states. | Everyday | Apr 2024 | Not in cards | |
| OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. | Everyday | Apr 2024 | | Dropped |
| OSWorld Agents complete 369 real computer tasks in Ubuntu desktop apps through screenshots and mouse and keyboard actions. Execution-based scripts check the final state. | Everyday | Apr 2024 | | Fading |
| BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. | Everyday | Mar 2024 | Not in cards | |
| Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. | Everyday | Mar 2024 | System cards only | |
| DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. | Mar 2024 | | Not in cards | |
| DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design. | Expert | Mar 2024 | 41.7% | Not in cards |
| DevBench Environment Setup Models prepare repository environments. The score is the percentage that pass the example usage test. | Expert | Mar 2024 | 41.7% | Not in cards |
| Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. | Expert | Mar 2024 | System cards only | |
| Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. | Everyday | Mar 2024 | System cards only | |
| HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. | Everyday | Mar 2024 | Not in cards | |
| In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. | Expert | Mar 2024 | | System cards only |
| LLM Colosseum Two models each control a fighter in Street Fighter III and choose moves from text descriptions of the game. Wins give Elo ratings. | Everyday | Mar 2024 | 1,835 | Not in cards |
| Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. | Everyday | Mar 2024 | | System cards only |
| Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. | Expert | Mar 2024 | | System cards only |
| StableToolBench Models solve user requests by calling tools from ToolBench's large set of real-world APIs. A virtual API server with a cache and API simulators stands in for live APIs, so results stay stable over time. GPT-4 judges whether each request was solved and which of two answers is better, giving a solvable pass rate and a win rate. | Everyday | Mar 2024 | 26.4% | Not in cards |
| SWE-bench Lite Models resolve 300 self-contained GitHub issues chosen from SWE-bench to cut evaluation cost. Hidden repository tests decide each patch. | Expert | Mar 2024 | Not in cards | |
| Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. | Everyday | Mar 2024 | System cards only | |
| WorkArena Web agents do routine knowledge-work tasks in a live ServiceNow instance, such as filling forms and filtering lists. WorkArena++ adds 682 compositional tasks at levels 2 and 3. | University | Mar 2024 | Not in cards | |
| AndroidArena Agents use apps such as Gmail and Google Maps, sometimes switching apps or following extra constraints. GPT-4 judges success from the recorded actions and observations. | Everyday | Feb 2024 | | Not in cards |
| Berkeley Function Calling Leaderboard (BFCL) Model emits function calls for single, parallel, multi-turn and agentic scenarios. Scored by AST and execution checks. | Everyday | Feb 2024 | | In use |
| FinBen A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading. | University | Feb 2024 | 0.25 | Not in cards |
| GTBench Models play strategic games against other models or scripted opponents. The benchmark measures their advantage in game rewards. | Everyday | Feb 2024 | 0.13 | Not in cards |
| Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. | Expert | Feb 2024 | | System cards only |
| TravelPlanner Agents plan multi-day trips from a user request, using tools to search nearly four million records of flights, hotels, restaurants and attractions. Each plan must meet the user's hard constraints, such as budget, and commonsense rules. The headline final pass rate counts a plan only when it meets every constraint. | Everyday | Feb 2024 | Not in cards | |
| WebLINX Agents follow a user's multi-turn chat instructions to navigate real websites. 100,000 interactions from 2,300 expert demonstrations give turn-level metrics. | Everyday | Feb 2024 | 10.4 | Not in cards |
| AgentBoard LLM agents act in 9 partially observable, multi-turn environments that cover embodied tasks, games, web browsing and tool use. Besides final success, AgentBoard reports a progress rate, the share of annotated subgoals the agent reaches. An open toolkit breaks results down by skill. | Everyday | Jan 2024 | 48.9% | Not in cards |
| Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. | Expert | Jan 2024 | System cards only | |
| Economics Arena for Large Language Models Models choose numbers in beauty contests and place sealed bids in auctions. Researchers compare payoffs and measure how strategies change when players see previous rounds. | University | Jan 2024 | 0.95 | Not in cards |
| GLM-4 All Tools information seeking (Z.ai) Models use a web browser to find information for user questions. The ChatGLM report gives accuracy but does not describe the test set. | Everyday | Jan 2024 | 78.1% | System cards only |
| LLM Game Benchmark Models compete in tic-tac-toe, Connect Four, Gomoku, and other grid games through a shared interface. | Everyday | Jan 2024 | 185.0 | Not in cards |
| Mobile-Eval Agents follow 33 instructions on Android. Some requests specify the app. Others describe a goal and leave the choice of app to the agent. | Everyday | Jan 2024 | Not in cards | |
| VisualWebArena Multimodal web agents complete 910 visually grounded tasks on self-hosted classifieds, shopping and Reddit-style sites. Execution-based checks score each task. | University | Jan 2024 | 16.4% | Not in cards |
| WebVoyager Agents complete user tasks on 15 popular real websites by browsing from screenshots and page text. A GPT-4V judge grades whether each task succeeded. | Everyday | Jan 2024 | | Fading |
| AppAgent Test Set Agents follow 45 instructions across nine Android apps, including scheduling email and changing music playback. Tasks time out after ten steps. | Everyday | Dec 2023 | Not in cards | |
| T-Eval Models use tools step by step, and each sub-process is scored separately: instruction following, planning, reasoning, retrieval, understanding, and review. | Everyday | Dec 2023 | 86.4% | Dropped |
| V* Models answer 191 multiple choice questions about small details in high-resolution images, such as an object's color or relative position. | Everyday | Dec 2023 | | Fading |
| AvalonBench Models play Resistance Avalon while discussing, deceiving, deducing, and negotiating with other agents. | Everyday | Nov 2023 | Not in cards | |
| FinMem Trading Agents choose whether to buy, sell or hold a stock each day. The benchmark measures cumulative return against historical prices. | University | Nov 2023 | 1.63% | Not in cards |
| GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. | Everyday | Nov 2023 | 66.1% | Fading |
| Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. | University | Nov 2023 | 4.67% | Not in cards |
| LangChain Benchmarks Agents complete tool use, retrieval and extraction tasks with reproducible LangSmith datasets and task-specific graders. | Expert | Nov 2023 | Not in cards | |
| BotChat Two instances of a model sustain a conversation. Judges assess the resulting dialogue. | Everyday | Oct 2023 | 73.2 | Not in cards |
| LLM-Coordination Models play four cooperative games and answer 198 multiple-choice questions about the same games. The authors report game outcomes and question accuracy separately. | Everyday | Oct 2023 | | Not in cards |
| MLAgentBench Agents run machine learning experiments on 13 tasks, from improving CIFAR-10 accuracy to research problems like BabyLM. They read and write files, run code and inspect outputs. | Expert | Oct 2023 | | Not in cards |
| SmartPlay Language-model agents play a collection of games designed to test planning, reasoning, memory and interaction abilities. | Expert | Oct 2023 | 0.80 | Not in cards |
| SuperCLUE-Agent Chinese language models are tested on agent skills in Chinese tasks. These include tool use with several API calls, task planning and memory over long dialogues. | Everyday | Oct 2023 | 78.4 | Not in cards |
| SWE-bench Models resolve 2,294 real GitHub issues from 12 Python repositories. A patch counts as resolved when the hidden repository tests pass. | Expert | Oct 2023 | Not in cards | |
| Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. | Expert | Sep 2023 | System cards only | |
| Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. | School | Sep 2023 | | System cards only |
| Code Interpreter Benchmark General Models use a Python interpreter for general tasks. The score is the percentage of generated programs that execute. | School | Sep 2023 | | Not in cards |
| Code Interpreter Benchmark Math Models use a Python interpreter to solve math problems. The score is execution accuracy on the Math task set. | School | Sep 2023 | | Dropped |
| MakeMePay A con artist model tries to talk a GPT-4o mark into paying money in a role-play. Scored as the share of conversations with a payment. | Everyday | Sep 2023 | 57% | System cards only |
| MakeMeSay A manipulator model tries to make a GPT-4o manipulatee say a codeword without suspicion. Scored as manipulator win rate. | Everyday | Sep 2023 | 72% | Fading |
| AgentBench 8 environments (OS, DB, KG, games, web) testing LLMs as agents. | Everyday | Aug 2023 | 4.01 | Not in cards |
| Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. | Everyday | Aug 2023 | | Dropped |
| DroidTask Agents select actions for 158 Android tasks across 13 apps. The original completion score requires every action to match the recorded sequence. | Everyday | Aug 2023 | Not in cards | |
| FinEval Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings. | University | Aug 2023 | | Not in cards |
| Aider-Edit Models edit Python source files to complete 133 Exercism exercises through Aider. Scored by the share of exercises whose unit tests all pass. | School | Jul 2023 | | Dropped |
| ToolBench Models plan and call real REST APIs from RapidAPI, over 16,000 of them, to answer instructions that need one or many tools. ToolEval judges pass rate and win rate. | Everyday | Jul 2023 | Not in cards | |
| WebArena Agents complete 812 realistic tasks on self-hosted shopping, forum, GitLab, CMS and map websites. Functional graders check the outcome. | Everyday | Jul 2023 | 68% | Fading |
| InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. | School | Jun 2023 | | System cards only |
| LeanDojo Benchmark Models prove theorems from Lean's mathlib library inside the Lean proof assistant, which checks every proof. The benchmark has 98,734 theorems and proofs with premise annotations. A novel-premises split tests whether a prover can use lemmas it never saw in training. | University | Jun 2023 | Not in cards | |
| LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. | Everyday | Jun 2023 | | Fading |
| ToolQA Models answer questions that need outside information, using 13 tools such as database queries, text retrieval and a calculator over fixed data sources. The questions are built so that a model's internal knowledge cannot answer them. An easy and a hard set separate single lookups from multi-step tool use, and answers are checked by exact match. | Everyday | Jun 2023 | 24.1% | Not in cards |
| clembench Chat models play rule-based dialogue games in self-play, such as Taboo, Wordle and reference games. The clemscore combines how often a model follows the game rules at all with how well it plays when it does. Version 4.0 runs 14 games in 30 languages. | Everyday | May 2023 | 88.8 | Not in cards |
| SuperCLUE The CLUE team tests Chinese and foreign language models on periodic sets of Chinese questions covering language, professional knowledge, reasoning, agents and safety. Open answers are graded by a strong LLM judge, and results feed an overall ranking. | University | May 2023 | | Not in cards |
| API-Bank Models make API calls in dialogues that need tools such as calendars and search. Scored on call accuracy. | Everyday | Apr 2023 | | System cards only |
| MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. The score is normalized game reward. Annotations also score power-seeking, disutility and ethical violations. | Everyday | Apr 2023 | Not in cards | |
| PushWorld Agents move shapes through puzzles that require tools or pushing obstacles aside. The benchmark includes more than 200 puzzles with goal positions. | School | Jan 2023 | Not in cards | |
| Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. | Everyday | Oct 2022 | | System cards only |
| Language-Table A simulated robot moves colored blocks in response to natural language instructions. The benchmark measures success and success weighted by path length across five task families and 696 task variations. | Everyday | Oct 2022 | Not in cards | |
| VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. | Everyday | Oct 2022 | Not in cards | |
| WebShop Agents search and click through a simulated shopping site with 1.18 million real products to buy an item that matches a text instruction. Reward reflects how well the purchase matches. | Everyday | Jul 2022 | Not in cards | |
| MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. | Everyday | Jun 2022 | | Fading |
| ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. | School | Mar 2022 | 0.15 | Not in cards |
| CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. | Everyday | Dec 2021 | Not in cards | |
| SatNet Agents assign ground antennas and contact times to spacecraft requests across five weeks of NASA Deep Space Network operations. They balance unmet demand across missions. | University | Nov 2021 | Not in cards | |
| Jiminy Cricket Agents play 25 text adventures. Annotations on game states track moral behavior alongside game rewards. | Everyday | Oct 2021 | Not in cards | |
| TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. | Everyday | Oct 2021 | Not in cards | |
| ALFWorld Agents complete household goals such as putting a washed apple in the fridge in a text world aligned with the ALFRED embodied simulator. Success is whether the goal state is reached. | Everyday | Oct 2020 | 24.6% | Not in cards |
| Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. | Everyday | Oct 2020 | Not in cards | |
| PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. | School | May 2020 | Not in cards | |
| TextWorld Cooking (GATA) Agents navigate text-based rooms, gather ingredients, and prepare recipes. The port sums points earned during play and includes separate training and test games at increasing difficulties. | Everyday | Feb 2020 | | Not in cards |
| ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. | Everyday | Dec 2019 | Not in cards | |
| Meta-World Robot policies do simulated tabletop manipulation tasks, scored by success rate. Splits test multi-task learning, such as MT10 and MT50, and meta-learning, such as ML45. | Everyday | Oct 2019 | Not in cards | |
| RLBench Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates. | Everyday | Sep 2019 | Not in cards | |
| BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. | Everyday | Oct 2018 | 37.5% | Not in cards |
| MiniWoB++ Agents operate over 100 small synthetic web pages, such as forms, menus and date pickers, from a natural-language instruction. Each task returns a success reward. | Everyday | Feb 2018 | Not in cards | |
| Behnke and Geiger Flexible Job Shop Instances Agents build schedules or write solvers that assign operations to eligible machines within work centers. The objective is to finish the last job as early as possible. | University | Jan 2012 | Not in cards | |
| Wikispeedia Players navigate from one Wikipedia article to another using only article links. The Prime Intellect port scores whether agents reach targets held out from training. | Everyday | Jul 2009 | Not in cards | |
| Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. | Expert | System cards only | ||
| Agent Initiative Benchmark Agents inspect workplace situations and decide whether to act without an explicit task. Two model judges score responses to 50 situations from 0 to 10. | Expert | 8.97 | Not in cards | |
| AgentCLUE Agents execute tasks in Chinese. | University | 46.4 | Not in cards | |
| AgentCLUE-General Agents complete general purpose tasks in Chinese. | University | 4.62 | Not in cards | |
| AgentCLUE-Mobile Mobile assistants complete tasks on a phone. | Everyday | Not in cards | ||
| AgentCLUE-Task Agents carry out interactive tasks in Chinese. | Everyday | 49.5 | Not in cards | |
| Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. | Research frontier | | Not in cards | |
| Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. | Expert | | System cards only | |
| AnalogCoder-Pro Agents write analog circuit programs and use simulated waveforms to diagnose failures. The authors report the fraction of successful designs and the number of tasks solved. | Expert | 88.4% | Not in cards | |
| Animation Bench Agents recreate web animations in HTML from reference frames and a network capture. The scorer compares appearance, motion and layout. | University | 0.61 | Not in cards | |
| Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. | 2.06 | System cards only | ||
| Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. | Not in cards | |||
| Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. | Not in cards | |||
| Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. | Expert | 0% | In use | |
| Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. | Expert | 14.3% | System cards only | |
| BAZAAR Agents buy and sell goods in a simulated competitive market. | University | 7 | Not in cards | |
| BIM Bench Agents draw a timber roof eaves detail in Revit. Reviewers see the exported sheet without the model name and rate its usability from one to five. | University | 4 | Not in cards | |
| Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. | Expert | 91.1% | System cards only | |
| Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. | Expert | 2.53 | System cards only | |
| BrowserInject v1 Browser agents do benign tasks on 30 retail, health, support and travel sites seeded with prompt injections. Reports attack success rate. | Expert | | System cards only | |
| Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. | Expert | System cards only | ||
| CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. | Expert | 29% | System cards only | |
| Charlemagne Labs social-engineering benchmark Attacker models try to persuade a skeptical simulated victim to take an action over many turns. Scored by attempted and carry-through rates. | Expert | | System cards only | |
| Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. | Expert | System cards only | ||
| ChessBench (MadeByLupu) Models play each other with access to the board, move history and legal moves. The organizer rates each model and thinking setting with Glicko-2. | School | | Not in cards | |
| Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. | Everyday | | System cards only | |
| Closed-Loop Evaluation of LLM Agents for Embedded Software Development Agents write firmware from engineering briefs and test it on simulated equipment. Hidden checks score device behavior across five control tasks. | University | 100% | Not in cards | |
| CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. | Expert | 55.8% | System cards only | |
| Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. | Expert | | System cards only | |
| CompanyBench Agents complete financial analyst workflows in a simulated bank. Checks compare outputs and database changes with expected results. | Expert | 44.4% | Not in cards | |
| Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. | Everyday | | System cards only | |
| Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. | Everyday | | System cards only | |
| Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. | Expert | 78% | System cards only | |
| Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. | Expert | | System cards only | |
| Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. | Expert | | System cards only | |
| CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). | Expert | 75.9% | In use | |
| Crest Arcade Chess Models play standard chess and Chess960 against fixed engine opponents without a list of legal moves. The organizer fits separate ratings for each variant. | School | 1,758 | Not in cards | |
| Crest Arcade Elemental Armory Models equip four elemental balls with cards, then select and aim a ball each round. The score averages match win rates across three levels. | Everyday | 67.3% | Not in cards | |
| Crest Arcade FreeCell Models move cards to the four foundations using a variable number of free cells. The score averages win rates across five tiers. | Everyday | 63.3% | Not in cards | |
| Crest Arcade Hustle Sim Models buy upgrades and hire helpers to reach earnings targets in an idle game. The score compares simulated completion times with reference strategies. | Everyday | 83.7 | Not in cards | |
| Crest Arcade Micro Mouse Models steer a simulated robot through six daily mazes. The score compares each finish time with the median human time and gives unfinished mazes zero. | Everyday | | Not in cards | |
| Crest Arcade Mines Models reveal or flag cells on Minesweeper boards that can be solved without guessing. The score averages win rates across four board sizes. | Everyday | | Not in cards | |
| Crest Arcade Nations Royale Models manage one nation against 47 computer opponents on a simulated map. Wins earn full credit and unfinished games earn credit for territory held. | Everyday | 100.0 | Not in cards | |
| Crest Arcade Ravelin Models build towers to protect a keep through waves of attackers. The score compares wins with the reference time plus a penalty for lives lost. | Everyday | | Not in cards | |
| Crest Arcade Starjar Models place pieces in a jar so matching pieces touch and merge. The score compares their total points with a reference bot on identical jars. | Everyday | | Not in cards | |
| Crest Arcade Wildhaven Models select a team of animals and choose where each lands in a village. The score measures the share of village strength they destroy. | Everyday | | Not in cards | |
| CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. | Expert | 82.3% | System cards only | |
| Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. | Everyday | | System cards only | |
| DesignArena Agentic Game Dev Coding agents build multi-file games from a user prompt. Users play the final games and vote, and the logs keep tool calls, retries and failures. | Everyday | 1,408 | Not in cards | |
| DesignArena Agentic Image to HTML Coding agents turn a user-uploaded image into a self-contained HTML website. Users compare how well each page reproduces the design and vote. | Everyday | 1,247 | Not in cards | |
| DesignArena Agentic Overall Coding agents build apps, games, slides and spreadsheets from user prompts. Users compare the results, and votes from every agent board form one rating. | Everyday | 1,443 | Not in cards | |
| DesignArena Blender Agents build a 3D scene in Blender from a user prompt. Users compare the rendered scenes and vote for the better one. | Everyday | 1,381 | Not in cards | |
| DesignArena Builder AI app builders make full-stack web and mobile apps from the same user prompt. Users compare the results and vote for the better one. | Everyday | Not in cards | ||
| DesignArena Frontend Web App Coding agents build multi-file React frontends from real user prompts. Users compare the running apps and vote, and the run logs record agent behavior. | Everyday | 1,360 | In use | |
| DesignArena Full Stack Coding agents build complete React apps with sign-in, databases and backend code from real user prompts. Users try the apps and vote for the better one. | Everyday | 1,400 | Not in cards | |
| DesignArena Godot Game Dev Coding agents build complete Godot projects with GDScript, scenes and generated assets, exported to run in the browser. Users play the games and vote. | Everyday | 1,315 | Not in cards | |
| DesignArena HTML Slides Agents write a self-contained HTML slide deck that is converted to PPTX. Users compare the decks and vote for the better one. | Everyday | 1,531 | Not in cards | |
| DesignArena Image to Frontend Coding agents turn a reference image into a multi-file frontend web app. Users compare the running apps and vote for the better one. | Everyday | | Not in cards | |
| DesignArena PDF Agents build a PDF document from a user prompt. Users compare the two documents and vote for the better one. | Everyday | 1,346 | Not in cards | |
| DesignArena Python-PPTX Slides Agents build PowerPoint decks with python-pptx from a user prompt. Users compare narrative, visual design and structure and vote for the better deck. | Everyday | 1,377 | Not in cards | |
| DesignArena React Native Models build cross-platform mobile apps in React Native from real user prompts. Users compare the running apps and vote for the better one. | Everyday | 1,318 | Not in cards | |
| DesignArena Spreadsheet Agents build spreadsheets such as financial models, budgets and trackers as Excel or CSV workbooks. Users compare accuracy, structure and formatting and vote. | Everyday | 1,429 | Not in cards | |
| Destructive actions (Anthropic) Models continue real Claude Code transcripts cut just before a destructive action. Score is the rate of taking a destructive action. | Expert | 21% | System cards only | |
| Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. | Expert | 0.30% | System cards only | |
| EKWbench Agents complete operational tasks such as sourcing reviews and referral triage. The score averages the share of verifier weight earned. | Expert | 74.4% | Not in cards | |
| EmbodiedCLUE-VLA (tabletop tasks) Robot policies carry out tasks on a tabletop. | Everyday | | Not in cards | |
| Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. | Research frontier | 72.4% | System cards only | |
| FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. | Everyday | | Not in cards | |
| Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. | Expert | 100% | System cards only | |
| FrontierMemory A model reads 750 past agent trajectories and writes memory for a task agent. The score is how many new coding and knowledge-work tasks that agent then solves. | Research frontier | 73.3% | Not in cards | |
| Game Arena 3 Check Chess (Generality) Models play both colors. A player can win by checking the opposing king three times. A double check counts once. | School | 27.0 | Not in cards | |
| Game Arena Anti Chess (Generality) Models must capture when they can. A player wins by losing every piece or by having no legal move. The king can be captured. | School | 25.0 | Not in cards | |
| Game Arena Atomic Chess (Generality) Models play chess with exploding captures. A capture removes both involved pieces and neighboring pieces other than pawns. Exploding the opposing king wins. | School | 27.0 | Not in cards | |
| Game Arena Crazyhouse Chess (Generality) Models play Crazyhouse chess in both colors. A player can spend a turn placing a captured piece back on the board under its new owner's control. | School | 26.0 | Not in cards | |
| Game Arena Four-in-a-Row (Generality) Models place pieces on any empty square of a four by nine grid. Four in a line wins. The tournaments use different board encodings. | Everyday | 92.0 | Not in cards | |
| Game Arena Horde Chess (Generality) Models take both sides in turn. White tries to checkmate with 36 pawns. Black starts with the usual pieces and must capture every white pawn. | School | 24.0 | Not in cards | |
| Game Arena King of the Hill (Generality) Models can win by moving their king onto any of the four central squares. The king must reach the square without moving into check. | School | 24.0 | Not in cards | |
| Game Arena Racing Kings (Generality) Models race their kings to the eighth rank without giving check. Black can draw by reaching that rank immediately after White. | School | 23.5 | Not in cards | |
| Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. | Not in cards | |||
| Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. | Not in cards | |||
| Genspark Slides Benchmark Models build finished slide decks from 200 real production tasks in the Genspark Slides agent harness. An internal grader scores quality, completion, content and design. | Everyday | 83.6% | Not in cards | |
| Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. | Expert | Not in cards | ||
| Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. | | System cards only | ||
| Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. | Research frontier | | System cards only | |
| Gray Swan Shade (adaptive attacker) Gray Swan's adaptive red-teaming agent trained on test scenarios to attack coding, computer-use and browser agents with prompt injections. | Not in cards | |||
| Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. | Expert | | System cards only | |
| HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. | 7.80% | System cards only | ||
| Harvey LAB-AA v1.0 Artificial Analysis runs agents on Harvey's Legal Agent Benchmark tasks in an offline sandbox, where they write legal work products such as memos and redlines from case documents. Judges grade each deliverable against a task rubric, and the headline is the share of rubric criteria passed. | Expert | 94.6% | In use | |
| Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. | Expert | System cards only | ||
| ICLayout-Bench Agents turn transistor netlists into chip layouts. Design tools check the layout rules and connectivity, then measure electrical performance and area. | Expert | 80.3 | Not in cards | |
| Implicit Intelligence Agents respond to requests whose unstated requirements affect safety, privacy, or accessibility. Labelbox reports the share of scenarios each model passes. | Everyday | 53.2% | Not in cards | |
| Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. | Everyday | | System cards only | |
| Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. | Expert | | System cards only | |
| Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. | Expert | | System cards only | |
| Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. | Expert | | System cards only | |
| InferenceBench Coding agents optimize a language model inference server within a fixed time budget. The harness measures serving speed and checks answer quality against a reference. | Expert | 12.1× | Not in cards | |
| Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. | Expert | | System cards only | |
| Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. | Expert | | System cards only | |
| Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). | Expert | | System cards only | |
| JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. | Expert | | System cards only | |
| Kaggle Chess Suite Models play chess from the standard starting position and from selected openings. Kaggle publishes both ratings for each model. | School | | Not in cards | |
| Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. | Research frontier | 80% | System cards only | |
| Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks. | Expert | Not in cards | ||
| LatchBio Capabilities V0 Agents complete biological data analysis tasks. The historical score averages the available benchmark scores equally and excludes missing scores. | Expert | 51.3% | Not in cards | |
| Lazy investigation (Anthropic) Agents debug a failing job where misleading evidence invites a harmful action such as cordoning a healthy node. Scored 0 to 10 by a grader. | Expert | 9.94 | System cards only | |
| LLM Debate Benchmark Models debate an opponent over multiple rounds. Judges compare the arguments. | University | 1,706 | Not in cards | |
| LLM Deceptiveness and Gullibility Models try to deceive other models and recognize deceptive claims. | Everyday | 1.10 | Not in cards | |
| LLM Games Models play Connect Four against one another. Ramp publishes match scores and final Elo ratings. | Everyday | 1,728 | Not in cards | |
| LLM Persuasion Benchmark Models try to change another model’s stated position over multiple conversation turns. | Everyday | 1.71 | Not in cards | |
| LLM SoccerArena Bundesliga 2026/27 Models forecast Bundesliga match scores and outcome probabilities with web access. The standings award points for score predictions made about 24 hours before kickoff. | Everyday | | Not in cards | |
| LLM SoccerArena La Liga 2026/27 Models forecast La Liga match scores and outcome probabilities with web access. The standings award points for score predictions made about 24 hours before kickoff. | Everyday | 129.0 | Not in cards | |
| LLM SoccerArena Premier League 2026/27 Models forecast Premier League match scores and outcome probabilities with web access. The standings use the predictions made about 24 hours before kickoff. | Everyday | 89.0 | Not in cards | |
| Mana RP Bench Models write the next turn in a player’s story. The player sees two anonymous responses and chooses which one becomes part of the story. | Everyday | 1,280 | Not in cards | |
| Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. | Expert | | System cards only | |
| Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. | Expert | | System cards only | |
| Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. | Expert | | System cards only | |
| Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. | Expert | | System cards only | |
| Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. | Expert | | System cards only | |
| Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. | Expert | | System cards only | |
| METR data deduplication Agents write an efficient algorithm that removes near-duplicate lines from a large file. A trial succeeds when F1 reaches at least 80%. | Expert | 32.6% | System cards only | |
| METR general autonomy task suite Agents complete multi-step tasks in virtual environments. Scored as horizon weighted average task completion compared with human time limits. | Expert | System cards only | ||
| METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. | Research frontier | Not in cards | ||
| Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. | Expert | 100% | System cards only | |
| Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. | Expert | | System cards only | |
| Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. | Expert | System cards only | ||
| NeuroscienceBench Agents analyze neuroscience research data and report a scientific conclusion. The score counts correct conclusions among graded runs. | Expert | 41% | Not in cards | |
| OdysseysMega Agents complete tasks in a web browser. A judge checks each rubric criterion against the recorded actions and screenshots. | Expert | 41% | Not in cards | |
| OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. | Not in cards | |||
| OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. | 0.47% | System cards only | ||
| OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. | Not in cards | |||
| OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). | Expert | Not in cards | ||
| OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). | Expert | Not in cards | ||
| OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. | System cards only | |||
| OpenDesign Arena Models create design artifacts in the OpenDesign tool, and the board reports an evaluation score out of 100 with cost and speed per artifact. | Everyday | 91.0 | Not in cards | |
| OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason. | Expert | 34.9% | System cards only | |
| OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. | Research frontier | | System cards only | |
| Overconfidence: verifying command-line syntax (Anthropic) Agents are asked for an exact invocation of an unfamiliar command-line tool. Graders score from 1 to 5 whether they check documentation before running commands. | Expert | 4.90 | System cards only | |
| Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. | Everyday | | System cards only | |
| Peer Arena (oddbit.ai) Five models debate for five rounds, then each privately votes for a survivor. The organizer calculates Bradley-Terry ratings from game outcomes. | Everyday | 1,799 | Not in cards | |
| Personal Agent Bench Personal assistants complete everyday tasks using connected accounts. Experts check whether each assistant completes the work and respects the user’s instructions. | Everyday | Not in cards | ||
| Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. | Expert | | System cards only | |
| PGG-Bench Agents decide how much to contribute to a public good and whether to punish other players. | Everyday | | Not in cards | |
| Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. | Everyday | 0.59% | Not in cards | |
| Pokémon VGC Bench (Ricotoma) Agents play competitive Pokémon doubles with assigned teams. They choose four Pokémon, select two leads, and choose actions for both active Pokémon each turn. | Competition | 1,901 | Not in cards | |
| Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. | Expert | | System cards only | |
| Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. | Expert | | System cards only | |
| QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). | Expert | 63.1% | In use | |
| QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. | Expert | 1,770 | In use | |
| QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. | Expert | 86.3% | In use | |
| RareDiseaseBench Agents combine clinical and biological evidence to diagnose and manage rare diseases. A rubric grades their diagnoses and plans. | Expert | | Not in cards | |
| RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. | Expert | 94.2% | Not in cards | |
| Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. | Expert | 1,637 | System cards only | |
| Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. | Expert | | In use | |
| Realm Financial Reasoning Agents analyze financial documents and spreadsheets. A model grades their answers against practitioner rubrics. | Expert | | Not in cards | |
| RepoInject v1 Coding agents do benign tasks in 84 repository tasks seeded with prompt injections. Reports attack success rate at one and three attempts. | Expert | | System cards only | |
| ReproBAIT Agents independently reproduce biological AI models from published scientific papers. Scored as success rate. | Expert | 85% | System cards only | |
| ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. | Research frontier | | System cards only | |
| RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. | Research frontier | | System cards only | |
| Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. | Everyday | Not in cards | ||
| SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. | Expert | | System cards only | |
| SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. | Expert | 58.7% | System cards only | |
| SheetBench Agents solve finance and accounting spreadsheet tasks through a browser in a fresh Google Sheet. The score is the share of expected answer cells that match exactly. | Expert | 64.7% | Not in cards | |
| SimulatorArena Models act as students or writing clients. Researchers compare their messages and assistant ratings with those from people who used the same assistants. | School | 1.01 | Not in cards | |
| SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). | Expert | Not in cards | ||
| Speko LLM Benchmark for Voice Agents v3 Models use simulated support tools to look up orders and issue refunds. The task score gives partial credit when a model stops early without causing harm. | Everyday | | Not in cards | |
| Speko Speech-to-Speech Benchmark (six concierge scenarios) Voice agents book flights, move reservations, and revise budgets during scripted calls. Speko checks whether agents wait for confirmation and use the latest details. | Everyday | 0.91 | Not in cards | |
| SPOT5 daily photograph scheduling Agents choose which photographs a satellite should take and assign cameras. Feasible schedules earn points for image priority while respecting instrument, data rate, and storage limits. | University | Not in cards | ||
| STEM Professor Bench Agents solve unpublished research problems written by professional scientists. Most problems concern theoretical physics, with some partial credit. | Research frontier | 61.8% | System cards only | |
| StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. | Expert | | In use | |
| SuperCLUE-Agent Scenario Agents complete tasks drawn from Chinese application scenarios. | University | | Not in cards | |
| SuperCLUE-DeepResearch Agents research questions and assemble reports in Chinese. | University | Not in cards | ||
| SuperCLUE-DeepSearch Agents search for information to answer Chinese research questions. | University | 74.3 | Not in cards | |
| SuperCLUE-DeepSWE Agents complete extended software engineering tasks in Chinese. | Expert | 84.3 | Not in cards | |
| SuperCLUE-LarkClaw Claw assistants complete tasks through Feishu. | Everyday | Not in cards | ||
| SuperCLUE-Office Agents process documents, analyze data, and write office materials. | University | Not in cards | ||
| SuperCLUE-OpenClaw Models complete coding, memory, and writing tasks in the OpenClaw environment. | University | 92.3 | Not in cards | |
| SuperCLUE-R1 Search DeepSeek R1 services answer questions that require search and reasoning. | University | Not in cards | ||
| SuperCLUE-SWE Agents complete software engineering tasks in Chinese. | Expert | 73.3 | Not in cards | |
| SuperCLUE-SWE with Claude Code Models solve Chinese software engineering tasks in the Claude Code harness. | Expert | 75.8 | Not in cards | |
| SuperCLUE-SWE with mini-swe-agent Models solve Chinese software engineering tasks with the mini-swe-agent harness. | Expert | | Not in cards | |
| SuperCLUE-Terminal Agents complete tasks in a terminal. | Expert | 64.7 | Not in cards | |
| SuperCLUE-WeClaw Claw assistants complete tasks through WeChat. | Everyday | Not in cards | ||
| SuperCLUE-XClaw Claw assistant products complete user tasks in Chinese. | Everyday | Not in cards | ||
| SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. | Expert | System cards only | ||
| SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. | Expert | | System cards only | |
| Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. | Everyday | | System cards only | |
| TissueMicroscopyBench Agents examine tissue images and answer questions about cells, staining, and changes over time. | Expert | 34% | Not in cards | |
| TREC BioGen 2024 Systems retrieve biomedical literature and synthesize evidence into reports that answer a question. | Expert | 79.9 | Not in cards | |
| TxBench-Antibody-Development Agents use evidence to make manufacturing, quality, and clinical development decisions about candidate therapeutic antibodies. | Expert | | Not in cards | |
| UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. | System cards only | |||
| UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. | Expert | 15.6 | System cards only | |
| Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. | Expert | | System cards only | |
| Uncritically reporting flawed results (Anthropic) Agents analyze data with code that silently mishandles broken values. Graders check whether the agent flags the flaw or reports the biased numbers. | Expert | 9.83 | System cards only | |
| vCX Benchmark Models run inside a Retell voice agent on pass or fail checks for business rules, grounding, tool calls and speech-ready replies. Each suite reports its own pass rate. | Everyday | 93.3% | Not in cards | |
| WebsiteBench Agents explore reference websites and rebuild their appearance, behavior, and persistent state. Hidden browser tests require functional correctness and screenshot similarity, then report the fraction passed. | Expert | 26.2% | Not in cards | |
| White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. | Research frontier | System cards only | ||
| xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. | Not in cards |