Benchmark Almanac

Agents

Models take a sequence of actions and decide when to stop, alone or with other agents. We count 2306 benchmarks here, 138 of them arenas. A benchmark belongs here when interaction is agentic or multi-agent.

How fast it grows

941 were released since July, more than the 535 in the first half of 2026.

  1. Earlier 57
  2. 2024 H1 60
  3. 2024 H2 77
  4. 2025 H1 186
  5. 2025 H2 253
  6. 2026 H1 535
  7. 2026 H2 941

How much is still open

No model has passed 85% on 995 of the 1232 benchmarks here that score in percent, 81% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
3D GameDev Arena Agents build playable 3D browser games from specifications and reference images. Players compare anonymous game pairs. The authors also test functionality and playability. Expert Nov 2026 99.8% Not in cards
3DArena-Modeling Agents create 3D models for 120 tasks covering precision and appearance. Tencent converts Elo ratings from blind comparisons into percentages. University Nov 2026 Not in cards
3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. Expert Nov 2026 59.5 Not in cards
AbMaze Agents make choices in a simulated antibody optimization experiment under a fixed budget. Recorded assay results determine normalized improvement on 200 test episodes. Expert Nov 2026 Not in cards
Accelerate Bench Coding agents speed up 32 real scientific applications, such as GROMACS and OpenFOAM, on a fixed CPU allocation within 8 hours. A hidden verifier checks correctness, and Score@1 averages the share of runtime saved, with incorrect programs scoring zero. Expert Nov 2026 67.4% Not in cards
AgenticFinBench Agents complete 230 financial requests in two languages. Graders apply task-specific rubrics and audit unsupported claims against external evidence. Expert Nov 2026 89.2 Not in cards
AgenticRL-Bench Agents debug and maintain reinforcement learning infrastructure. A task passes when the implementation meets every hidden check. Expert Nov 2026 22.8% Not in cards
AgentScript-Eval Models write, debug, or optimize AgentScript definitions for Salesforce agents. A task passes when the program compiles and passes every execution test. University Nov 2026 37% Not in cards
ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. Everyday Nov 2026 80.4% Not in cards
AlphaSense Financial Research Benchmark Agents answer 245 multi-step finance questions across company research, financial modeling, earnings, deals, markets and regulatory analysis using source retrieval. Expert Nov 2026 Not in cards
ARE-bench Agents continue an existing research project from a real user request. A judge checks the deliverable against the request and earlier workspace requirements. Expert Nov 2026 66.2% Not in cards
AscendGYM Agents optimize existing kernels for Ascend processors using tensor shapes from real workloads. The benchmark measures speedup and token consumption. Expert Nov 2026 74.0 Not in cards
ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. Everyday Nov 2026 Not in cards
AstroMemBench Spacecraft agents use information from earlier observations to complete simulated missions. The environment scores task completion, safety, and resource use. University Nov 2026 Not in cards
AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. University Nov 2026 Not in cards
Autoform-Hard Agents complete Lean 4 proofs on 56 formalization tasks that earlier models failed. Expert Nov 2026 87.5% Not in cards
Backrooms: 3D Clue-Trail Navigation Agents follow visual clues through a first-person maze. Only an escape without a wrong doorway counts as a solve. Everyday Nov 2026 Not in cards
Beyond Survival Models play social deduction games with human-aligned strategies and interaction patterns. Everyday Nov 2026 58.7% Not in cards
BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. Expert Nov 2026 66.7% Not in cards
Browser Use LinkedIn Agents answer plain-language questions about LinkedIn using recorded page snapshots. The leaderboard reports correctness, cost, and time per task. Everyday Nov 2026 97.9 Not in cards
Clocktower Bench Models play the social deduction game Blood on the Clocktower (the Trouble Brewing script) against each other, with a scripted Storyteller. One model takes every good seat and another every evil seat, so each game tests one model's detection of the Demon against another's deception. A Bradley–Terry fit turns 386 games into Elo ratings. Everyday Nov 2026 1,859 Not in cards
Collaborative Intelligence Benchmark A model works in a team with scripted teammates and live model partners, and a teammate creates a test: pressure to take an unsafe shortcut, misconduct to report, a need nobody states, a conflict, or a cheat that a weaker model left behind. The 240 tasks across 8 dimensions of teamwork come from real records, such as aviation accident reports and clinical cases, and from existing benchmarks, such as SWE-bench and negotiation datasets. Each task has a checkable outcome. A model is scored only on the tasks it solves alone, so the score measures teamwork, not ability. Everyday Nov 2026 82.2% Not in cards
Company Agent Bench 205 questions on how well an agent understands its company, built from a month of real Slack history: who wrote an anonymized message, whom a teammate tagged, and who sent each reply. Agents search a replayed sandbox workspace using only messages from before the answer. Everyday Nov 2026 82.2% Not in cards
Construction AI Leaderboard Models answer construction questions, inspect drawings, edit building models, and generate validation rules. ONESTRUCTION averages the representative scores from five evaluations. University Nov 2026 73.7 Not in cards
Coverage Cat AI Insurance Benchmark (fictional insurance tasks) Models read fictional policy documents, find missing facts, and complete insurance applications. The scorer checks source quotes, required fields, and authorized actions. University Nov 2026 100% Not in cards
Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Expert Nov 2026 8.30% In use
Crazyrouter AI Trading Arena Models buy, sell, or hold three cryptocurrencies every five minutes using simulated accounts. The grader values portfolios at live quotes and deducts trading fees. University Nov 2026 -7.45% Not in cards
CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. Everyday Nov 2026 0.56 Not in cards
CursorBench 4.1 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported. Expert Nov 2026 65.4% In use
CWE-bench 2 Agents audit and patch vulnerabilities in 250 codebases. The score is pass@1 under deterministic verification. Expert Nov 2026 75% In use
Databricks AI Index Coding agents complete tasks at different costs. The index compares cost per task and solution quality alongside enterprise usage statistics. Expert Nov 2026 96.0 Not in cards
DeepFinance Agents do private equity and investment banking work on 21 anonymized real deals: valuation, due diligence, underwriting and financial modeling. Each task comes with a data room of about 19 files and would take a professional about 27 hours. Rubrics of about 84 criteria grade the output, the workflow and the judgment behind each assumption. Expert Nov 2026 62% Not in cards
DeGennaro Workplace sample Agents act as an employee’s assistant using 80 simulated workplace tools and anonymized company records. Four tasks grade the final state. Expert Nov 2026 Not in cards
DeputyBench Agents handle business requests under paired permission settings. The evaluator checks whether they act when authorized and seek evidence or refrain when authorization is missing. University Nov 2026 92.6% Not in cards
Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Expert Nov 2026 4.70% In use
Deta Studio AI Construction Takeoff Models use Deta’s tools to mark floor areas and count openings and electrical devices on four drawings. The rating combines accuracy, cost, and time. University Nov 2026 76 Not in cards
DrivingBench v2 Vision-language models drive a real Tesla through two unseen cone courses with steering, accelerator and brake tools. The score is the furthest progress over three attempts in one conversation, averaged over six trials. Everyday Nov 2026 89.5% Not in cards
EarthRecord Agents analyze multidecade satellite observations to answer land-change questions. Reference checks score answers to 238 questions across 56 sites. Expert Nov 2026 85.4% Not in cards
Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. University Nov 2026 Not in cards
EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. University Nov 2026 Not in cards
Enterprise-Bench L1-L2 v2 Agents answer 42 operational questions using six enterprise services. A GPT-6 Luna judge evaluates the final answer against task criteria. University Nov 2026 65.7% Not in cards
EnterpriseOps-Gym 1.1 Agents complete 296 workflows across customer service, human resources, and IT service management. SQL checks grade the resulting database state. Expert Nov 2026 Not in cards
EUREKA-Bench (OpenAI) Agents build predictive models for six biology research tasks. Scored as predictive performance with the best published predictor in each field set to 100. Research frontier Nov 2026 100.3 System cards only
ExcelFixBench Agents repair formulas, data, and formatting in an existing workbook. The task rubric checks the repair and whether unrelated content survived. University Nov 2026 27.8 Not in cards
ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. Everyday Nov 2026 Not in cards
Fermat's Last Theorem in Lean (Anthropic) Agent teams formalize Fermat's Last Theorem in Lean. Scores measure the weighted proportion of 60 accepted intermediate theorems. Research frontier Nov 2026 35% System cards only
FinDesk-Bench Agents find Treasury figures and use them to handle client requests across ten business services. A model grades API calls and the records agents leave behind. University Nov 2026 90.4% Not in cards
Firecrawl Search Bench Agents answer hard research questions with web search. A deterministic grader checks each answer against reference facts. Expert Nov 2026 79% Not in cards
FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. University Nov 2026 74.5% Not in cards
FraudBench (GOAL Lab) Models face 159 scenarios drawn from 112 real U.S. fraud lawsuits. Yes or no questions, open questions and agent tasks test whether a model recommends, assists or commits fraud. University Nov 2026 10.1% Not in cards
Frontier3DBench Agents create editable Blender objects and scenes or executable Unreal Engine worlds across 145 tasks and nine task families. Model judgments and native project checks assess appearance, geometry, requirements, and interactions. The overall score weights object, scene, and world scores by 30%, 30%, and 40%. Expert Nov 2026 66.0 Not in cards
FrontierBioML 25 machine learning tasks in computational biology, across antibody development, protein engineering, structural biology, binder design and functional genomics. An agent gets real experimental data and a brief, works offline for up to six hours and submits a prediction script. A sealed grader runs the script on held-out data, such as new antibody families or new protein targets, and scores each task from 0 at a simple baseline to 100 for a perfect answer. Expert Nov 2026 40.3 Not in cards
FrontierBreach Agents perform simulated enterprise tasks while processing untrusted material. The benchmark checks whether each agent resists instructions that conflict with its assigned task. Expert Nov 2026 2.40% Not in cards
FrontierEE Bench Agents run electrical engineering experiments on real lab hardware through networked instruments, camera photos and tickets to a human operator. Deterministic verifiers check 10 tasks against protected bench records. Expert Nov 2026 36.7% Not in cards
FrontierML Agents spend a fixed budget on GPUs and model APIs to train models for 21 tasks. An offline evaluator grades their submissions on held-out data. Research frontier Nov 2026 58.9 Not in cards
FrontierRSI Agents do 27 open-ended AI research and engineering tasks across pre-training, post-training and harnesses, inference serving and GPU kernels. Each trial is scored against the best valid result any model reached. Research frontier Nov 2026 34.3% Not in cards
GDPval v2 (OpenAI) Multistep assignments drawn from real projects in consulting, finance and design, given with the files and messages the team had when the work was assigned. OpenAI has not published the task count or grading. Expert Nov 2026 50.4% In use
GDPWorks Agents complete 83 professional tasks across 36 occupations, using source files to produce work such as documents, spreadsheets, and code. The Joules-to-GDP framework compares outputs with expert deliverables through a blinded model judge. Task success measures the share of wins and ties against the expert. Expert Nov 2026 Not in cards
GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. Expert Nov 2026 16.4% Not in cards
GeoGuesser Bench Agents explore offline Street View panoramas to report coordinates, name places, and find objects. Coordinate guesses earn distance points. Other answers require an exact match. Everyday Nov 2026 22.8 Not in cards
GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. Everyday Nov 2026 20.7% Not in cards
GroupAssistBench Assistants use memories from group conversations to answer questions and complete real-world group assistance tasks. Model judges score grounded helpfulness. Expert Nov 2026 Not in cards
Hard12 Agents solve 12 selected coding tasks from five benchmarks. Hidden tests grade each task. The leaderboard averages three runs per task. Expert Nov 2026 83.3% Not in cards
Harness Leaderboard Coding-agent harnesses such as Claude Code, Codex and OpenCode run the same open model at the same reasoning effort on 198 tasks from Terminal-Bench 2.1 and DeepSWE v1.1. Scored as the resolve rate averaged across the two benchmarks over three attempts, with any access to the answer counted as a fail. Expert Nov 2026 Not in cards
HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. Everyday Nov 2026 18.8% Not in cards
Historical Systems Bugs Agents repair two historical C bugs in Linux and Unikraft. Hidden tests exercise the patched code, including driver behavior under emulated hardware. Expert Nov 2026 Not in cards
HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. Everyday Nov 2026 61.9% Not in cards
HoneyBench v0.2-alpha Measures reward hacking: how often an agent games its grader instead of doing the task. Lower is better. Research frontier Nov 2026 44.0 In use
Honor of Kings Bug AI Root Cause Analysis Agents use the Honor of Kings Bug AI platform to identify the cause of code defects. The team measures how often agents identify the correct cause. University Nov 2026 73.8% Not in cards
Honor of Kings Game Configuration Agents configure game data with the Honor of Kings spreadsheet platform. The team measures configuration accuracy before and after the agent checks and repairs its work. University Nov 2026 Not in cards
How frontier models miss risk in routine enterprise work Agents handle workplace requests whose risks appear in the prompt or in supporting files. Judges check whether the agent completes an unsafe action. Expert Nov 2026 Not in cards
HW-Bench Agents complete six hardware engineering tasks for a 2.45 GHz antenna using native design and simulation tools. Code checks and calibrated model judges grade outputs against engineer-written criteria. The headline pass rate measures runs that satisfy every critical criterion, averaged across tasks. Expert Nov 2026 67% Not in cards
HWE-Bench v1.1 (RTL bug repair) Agents fix hardware bugs in 417 repository tasks. The revised tests check whether each fix restores the required behavior. Expert Nov 2026 99% Not in cards
Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. Expert Nov 2026 83.9% Not in cards
Infrathon Agents optimize AI infrastructure under correctness and fidelity constraints. Hidden workloads measure performance gains, with additional checks for rule compliance. Expert Nov 2026 91.2 Not in cards
InSAR Regional Snow Mapping (Axis Spatial) Agents analyze Alpine radar scenes, align weather observations with radar windows, and calculate regional statistics. A script supplies reference answers for 24 tasks. University Nov 2026 100% Not in cards
Invoice Approval (Ali Siala) Agents process 60 supplier invoice work orders under a German company’s approval policy. The evaluator compares the resulting database with a reference state. University Nov 2026 Not in cards
Joi-Bench-Task Voice agents use tools while a simulated caller changes a request or interrupts a reply. The evaluator tracks completion, responsiveness, and omitted information. Everyday Nov 2026 Not in cards
Kaeru Flip (paprika3d) Agents play an 86-note rhythm game from browser pixels. They must finish the song with at most one miss and leave autoplay off. Everyday Nov 2026 Not in cards
KradleBench Models play live multi-agent Minecraft games and act by writing code. The overall score weights collaboration, economy and conflict at 30% each, and logic and creativity at 5% each. Everyday Nov 2026 78.3 Not in cards
LeanLean Agents shorten Lean projects while preserving protected theorem statements and passing the build and axiom checks. Expert Nov 2026 50.5% Not in cards
Legal Information Retrieval Benchmark Search APIs retrieve the official document behind legal research requests. Exact URL matching scores whether the official page appears first or in the top ten. Expert Nov 2026 Not in cards
Legal World sample Agents act as a legal assistant in an anonymized law practice. Four tasks grade the drafts, mail, calendar changes, and files they leave behind. Expert Nov 2026 Not in cards
LevelBench-100 v5.0 Models use ledger exports and supporting workbooks to close small business accounts. They reconcile transactions, apply accounting rules, and identify missing records. Expert Nov 2026 95.2% Not in cards
LifeVal Assistants carry out everyday personal tasks rather than advise on them: finding childcare, booking a car inspection, ordering groceries on a budget, handling government paperwork. Tasks come from time-use surveys and from about 100 people who recorded tasks they wanted an assistant to handle. Score is average reward across tasks. Everyday Nov 2026 86.1% In use
Longtail-SWE Agents resolve 202 GitHub issues across 13 programming languages with limited benchmark coverage. Hidden tests check the fix and verify that previously passing behavior still works. Expert Nov 2026 81.5% Not in cards
Malicious agentic influence campaigns, 34-point rubric (Anthropic) Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale. Expert Nov 2026 System cards only
ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. Everyday Nov 2026 Not in cards
MdCoEditBench Agents edit selected Markdown in the WorkBuddy editor alongside a user. Tasks include restoring a version, resolving an edit conflict, and continuing existing text. University Nov 2026 Not in cards
MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. Everyday Nov 2026 50.4% Not in cards
METR Capability Index METR fits an item response model to agent results on Time Horizon 1.1, TH2 and MirrorCode. The index places models on a common ability scale. Research frontier Nov 2026 859.0 Not in cards
METR Time Horizon 2 Agents complete 36 tasks across 12 families. METR estimates how long a skilled human would need for tasks that agents complete with 50 percent reliability. Research frontier Nov 2026 27.1 h Not in cards
MindGames Agents compete in Colonel Blotto, Iterated Prisoner’s Dilemma, Codenames, and Secret Mafia. Everyday Nov 2026 Not in cards
Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Expert Nov 2026 9.80% In use
MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. Expert Nov 2026 56.5% In use
Moneda Bench Models answer finance questions across seven separate benchmarks. AdviceBench, CiteBench, FilingsBench, GroundBench and PortfolioBench have their own records. KnowledgeBench and MarketBench have no published results yet. Moneda publishes no combined score. Expert Nov 2026 Not in cards
Moneda GroundBench Models answer live questions about market figures. Moneda’s default board reports the share of correct answers against reference data, with incorrect answers reported separately. Expert Nov 2026 Not in cards
MTGBench Models play multi-turn games that test reasoning, planning, and interaction across varied game types. Everyday Nov 2026 Not in cards
Multi Agent Coordination Bench Teams of eight agents work for four hours on twelve tasks built so each agent depends on its teammates. Each model also runs alone and as a team that can't talk, which isolates the gain from coordination. Expert Nov 2026 0.53 Not in cards
Nim compiler repair benchmark Agents repair the Nim compiler or standard library from linked issues. The current README lists 285 tasks graded with compile and regression tests. Expert Nov 2026 Not in cards
Nim Hard Agents fix Nim compiler issues on Linux. Black-box tests grade changes to the C backend. The current README lists 15 tasks. Expert Nov 2026 Not in cards
OfficePrism Agents resolve Chinese workplace tickets using one business system at a time. The evaluator checks outcomes across 420 tasks in seven business functions. University Nov 2026 76.4% Not in cards
On Evaluating User Simulators for Interactive Agents Models play the user in replays of 351 real sessions. The evaluation compares simulated behavior with observed behavior and measures agreement with human rankings of nine agents. Everyday Nov 2026 Not in cards
OWE-Bench 26 long-horizon engineering and research projects across software, physical engineering, machine learning and scientific computing, from GPU kernels to aircraft design. Agents get 4 to 18 hours per attempt, and an offline verifier scores what they ship on held-out cases. Research frontier Nov 2026 67.2% Not in cards
Parley Labs negotiation tasks Agents negotiate contracts and plan around market changes with scripted counterparties. An offline verifier grades outcomes. Ten public development tasks accompany private scored sets. Expert Nov 2026 Not in cards
Portal2Bench Agents solve Portal 2 chambers using screenshots, position readings, and game controls. Each chamber counts if the agent reaches the next map within $30 and 300 plans. Everyday Nov 2026 26.0 Not in cards
PPTCoEditBench Agents edit selected content in PowerPoint slides. Tasks include changing a title or table while preserving surrounding shapes and animations. School Nov 2026 Not in cards
PPTFixBench Agents inspect a PowerPoint file and repair its formatting and content. Each of the 50 files contains more than 25 issues. University Nov 2026 Not in cards
PReMind-Bench Models inspect a scene and decide whether to retrieve a personal memory before answering. Judges check whether the answer addresses the hazard and uses the right person’s information. Everyday Nov 2026 87.3% Not in cards
Prompt Injection in Benchmark Environments An automated attacker plants prompt injections in 115 held-out agent environments adapted from Gray Swan challenges. Scored on the share of attacks the model resists. Expert Nov 2026 99.8% System cards only
Prompt Injection in Model-Generated Environments An automated attacker writes both the agent environments and the prompt injections in them. Scored on the share of attacks the model resists. Expert Nov 2026 91.7% System cards only
PWS-Decisions Decision APIs choose answers to 300 closed-ended questions whose answers depend on current or scattered facts on the public web. Expert Nov 2026 Not in cards
PWS-Health Web research systems answer healthcare questions from clinical trials, medical literature, registries, drug labels, and healthcare workflows. Expert Nov 2026 Not in cards
PWS-Legal Web research systems answer dated questions about companies, laws, regulations, intellectual property, court decisions, and legal documents. Expert Nov 2026 Not in cards
Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. Expert Nov 2026 61.1% Not in cards
Ramp SWE-Bench v2 Agents solve 500 engineering tasks from Ramp's production backend. A panel of model judges compares each patch with the code engineers merged. Expert Nov 2026 76.4% Not in cards
Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. Expert Nov 2026 0.60 Not in cards
Reasoning extraction red-teaming Six attacker models try to make a model reveal its hidden reasoning. Scored on how often an attack succeeds. Expert Nov 2026 System cards only
RedstoneBench Agents design redstone circuits in Minecraft. A game server tests whether each circuit works and measures the space it occupies. Expert Nov 2026 88.7 Not in cards
RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. Everyday Nov 2026 51.4% Not in cards
RoboCasa-Mem Robots complete nine household tasks that require memory. Later steps depend on objects the robot saw or actions it took earlier. Everyday Nov 2026 16% Not in cards
RoboColosseum Physical robots execute fresh instructions in paired blind trials. Human evaluators choose the better run or a tie, and comparisons determine ratings for each embodiment. Everyday Nov 2026 Not in cards
RoboDx Agents repair faults in robot navigation software. Hidden driving scenarios test the repair. A separate diagnosis track asks models to identify faults without seeing the code. Expert Nov 2026 100% Not in cards
RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. University Nov 2026 80.6% Not in cards
RollerCoasterTycoonBench Models manage a theme park by choosing weekly actions in RollerCoaster Tycoon 2. Everyday Nov 2026 Not in cards
Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. Research frontier Nov 2026 44.7 Not in cards
Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. Everyday Nov 2026 42.9% Not in cards
Sequitur Research Lab Automation Agents run simulated wet-lab automation protocols. Scored out of 100 on how well each experimental phase is carried out, averaged over six runs. Research frontier Nov 2026 74.5 System cards only
ShadowBench Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match. Everyday Nov 2026 48.8 Not in cards
SMB Bench An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule. University Nov 2026 $7,749 Not in cards
SolidWorks Bench Agents model the custom parts of 73 real machines in SolidWorks and assemble them from raster drawing packages, through screenshots and the SolidWorks API. A grader checks the geometry against the reference and reads mates, motion and feature trees from the native files. Expert Nov 2026 0.57 Not in cards
SpaceChem Bench Agents program reactors to assemble molecules in SpaceChem. The main ranking measures how close their solutions come to the community’s records. Competition Nov 2026 74.3 Not in cards
Spatial Agent Index v2 Agents analyze geospatial files and produce numbers, maps, or lists. Axis Spatial averages the scores for each included evaluation. University Nov 2026 86.0 Not in cards
Spatial Intelligence Index v2 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. University Nov 2026 88.7 Not in cards
SWE Atlas v1.1 Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. Expert Nov 2026 80.5% Not in cards
SWE-CoWork Coding agents join a simulated engineering team on 101 real repositories and must gather scattered requirements from simulated colleagues and workplace records before implementing them. Hidden tests score the share of new-behavior tests passed and whether the whole task passes. Expert Nov 2026 7.92% Not in cards
SWE-Pair Agents fix two real bugs from the same Python project using one combined issue report. Hidden tests from both pull requests grade each fix. Expert Nov 2026 90.4% Not in cards
TCSExam Agents prove theorems, write solvers, build mathematical constructions and address open conjectures in theoretical computer science. Research frontier Nov 2026 71.9 Not in cards
TCSExam-Lite Agents solve 20 theoretical computer science tasks, five per track. The default solver protocol gives hidden feedback only after submission. Research frontier Nov 2026 72.3 Not in cards
Terminal-Bench 4.1 The Terminal-Bench 4.0 tasks with hardened verifiers: agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. Expert Nov 2026 71.2% In use
Terminal-Bench Math Modeling Agents use a terminal to solve 18 mathematical modeling problems and submit numerical answers. Feasibility checks and numerical scoring compare answers with reference solutions. Competition Nov 2026 64.3 Not in cards
TIS-100 Bench Agents write and optimize assembly programs for 55 puzzles. The ranking counts puzzles where a program improves on the community’s recorded tradeoffs in cycles, nodes, and instructions. Competition Nov 2026 11.0 Not in cards
Toloka Arena Enterprise Tool Use (November 2026) Agents complete private workflows across 11 industries. They must leave the right records and follow the required process in all five attempts. Expert Nov 2026 81% Not in cards
TriPhysBench Models answer quantitative questions about physical events in videos. Some questions require measuring an object or looking up its specifications. School Nov 2026 50.4% Not in cards
UAV Robust Control Hub Agents construct policies for a fictional civilian drone simulator. The benchmark evaluates policies on hidden scenarios and reports separate results under changed conditions. University Nov 2026 Not in cards
UK AISI File Download monitor evasion The model tries to download a forbidden file without a monitoring model noticing. Scored on how often it evades the monitor. Expert Nov 2026 System cards only
UK AISI simulated sandbox escape monitor evasion The model tries to escape a deliberately vulnerable sandbox without a monitoring model noticing. Scored on the share of samples where it escapes undetected. Expert Nov 2026 0.50% System cards only
VABAR Agents program robots to sort tools, mesh gears, and handle cables in simulation. The harness checks success as object shapes and positions vary. University Nov 2026 Not in cards
VAHBench Agents turn short drama briefs into finished videos with specified characters, dialogue, and durations. The evaluator checks completeness, quality, reliability, and accuracy of production reports. University Nov 2026 51.8 Not in cards
Vals Inference Provider Index Agents use different inference providers to complete the same finance, coding, and legal tasks. Expert Nov 2026 71.2% Not in cards
Vals Smith: pingdotgg/t3code Agents modify pingdotgg/t3code to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Nov 2026 Not in cards
Veris FinAgent Arena v1.1 Agents investigate bank alerts, read Treasury scans, and complete research requests. The index combines accuracy and repeatability across three benchmarks, then applies a cost penalty. University Nov 2026 757.0 Not in cards
Vibe Code Bench 1-100 v1.1 Agents extend a working web application across successive requests while preserving earlier features. Expert Nov 2026 55.2% Not in cards
VibePrevisBench Coding agents write Blender Python that turns a script, photos, an opening frame or reference footage into a graybox 4D previs scene and render it. Rule checks score the exported scene against hidden facts, and a VLM judge checks that those facts are visible in the render. Expert Nov 2026 77.7 Not in cards
VideoACE Models answer questions about fast actions in short videos. They can inspect selected frames more closely and request hints from a simulated viewer. Everyday Nov 2026 88.1% Not in cards
Voice Agent LLM Benchmark Models handle simulated caller requests in text. Hamming grades task completion, policy compliance, and how naturally the models phrase their replies. Everyday Nov 2026 Not in cards
Wayfinder Code Search Benchmark Coding search agents find the files changed by real bug fixes from short commit subjects or longer problem briefs. Expert Nov 2026 Not in cards
WeChat Pay Physical Client Evaluation Agents operate the WeChat Pay client on physical phones. The team scores execution paths, use of supplied knowledge, acceptance checks, and account creation accuracy. University Nov 2026 Not in cards
WordEditBench Agents edit Word documents. A rule-based checker reads the document XML, rewards requested changes, and penalizes changes to unrelated content. University Nov 2026 57.2 Not in cards
Work360 Agents complete professional tasks drawn from a map of 26 industries, 182 subfields and 721 specialties. Qwen in-house scoring (method unpublished). Expert Nov 2026 52.7 In use
WorkBuddy User Blind Evaluation Agents complete everyday work requests. WorkBuddy users rate their satisfaction on a four-point scale without seeing the model identity. University Nov 2026 3 Not in cards
WorkflowArena Agents write executable workflows that coordinate other agents on professional tasks. Hidden verifiers check the completed deliverables. Expert Nov 2026 37% Not in cards
WorkIndex Agents complete everyday WorkBuddy tasks. Tencent combines eight tests covering office files, audio and video work, and visual questions. University Nov 2026 Not in cards
Yet Another Kernel Kernel Models optimize GPU kernels for speed while a verifier rejects numerical errors and input-insensitive shortcuts. Research frontier Nov 2026 Not in cards
4DCodeBench Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion. Expert Oct 2026 0.79 Not in cards
AA-3D Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings. Expert Oct 2026 1,318 In use
AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. Everyday Oct 2026 Not in cards
AAArena Agents write and revise programs for 12 games. Each program competes against archived human entries under a fixed match budget. University Oct 2026 6.00 Not in cards
AdaExploreBench Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice. Everyday Oct 2026 70.3% Not in cards
Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. Everyday Oct 2026 91.4% System cards only
Agent4Sci Coding agents try to improve the published best model on 20 scientific ML benchmarks across 11 domains, under long in-the-wild runs. The paper reports mean relative improvement rather than a ranking. Research frontier Oct 2026 20.1% Not in cards
AgentCallBench: Tool-Calling Reliability Models operate a simulated store using tools. The grader checks tool choice, arguments, failure recovery, unnecessary calls, and refusals of impossible orders. Everyday Oct 2026 0.99 Not in cards
Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. Expert Oct 2026 47% System cards only
Agentic Credit Analysis Models extract and standardize income statements and balance sheets. Ramp compares their figures with credit analysts’ figures using a five percent tolerance. Expert Oct 2026 80.2% Not in cards
Agents' Last Exam v1.1 Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references. Expert Oct 2026 47.5% In use
AgentSWE Coding agents build, edit, or optimize agents across 25 software-engineering commissions. Frozen deliveries run on held-out cases, where execution gates and task-specific model-judged rubrics score behavior. Optimization tasks measure improvement against each source benchmark objective. Expert Oct 2026 19.1 Not in cards
AGORA (Agentic Global Open Research Arena) Autonomous AI systems get one secret open finance question, 72 hours and a fixed budget to write a paper. Five finance scholars judge it. Research frontier Oct 2026 Not in cards
Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. Everyday Oct 2026 87.9 Not in cards
AML Bench (Veris AI) Agents investigate suspicious activity at a simulated bank. A model judge checks whether the recorded service requests and final case disposition satisfy every requirement. Expert Oct 2026 70.8% Not in cards
Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. Expert Oct 2026 Not in cards
ArenaWebdevCorrectness Judges inspect generated applications and predict whether their features work. Human labels provide the reference answers. University Oct 2026 81.6% Not in cards
Argo-Bench Agents query a simulated food delivery company's warehouse and act on their findings. The grader measures the business consequences of their decisions across 210 tasks. Expert Oct 2026 70.2 Not in cards
Artificial Analysis Coding Agent Index v2 Coding agents, each a model plus harness, attempt software engineering, terminal, and repository question tasks. The index averages three equal-weight benchmark scores. Oct 2026 58.6 In use
Artificial Analysis Intelligence Index v5 Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Oct 2026 64.6 In use
ASTRA-bench Agents use personal context and interactive tools to complete more than 2,000 multi-step scenarios. Automated checks score whether each user intent is satisfied. Expert Oct 2026 Not in cards
ATLAS (Exa) Agents search the web to build a full table of every entity that meets a set of conditions, with 2 to 10 attributes each. A row counts only if every cell is correct. Expert Oct 2026 Not in cards
ATLAS-Finance v1.1.0 Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level. Expert Oct 2026 10.7% Not in cards
AutoFactory-Bench Agents write control programs for simulated factory cells. A run passes when the program completes every operating condition without a safety violation. University Oct 2026 72.2% Not in cards
AutoformBench Agents fill in missing Lean proofs for mathematical statements and submit their solutions to a clean grading workspace. Expert Oct 2026 82.4% Not in cards
AutoVend Agents buy stock and schedule deliveries for 20 simulated vending machines over 180 days. They share one warehouse and pay for each site visit. University Oct 2026 309.4 Not in cards
BallotBench Nine frontier agents forecast 863 contests in the 2026 US midterms each day until polls close. Proper scoring rules grade forecasts against certified results, beside Kalshi, Polymarket and expert ratings. Expert Oct 2026 0.11 Not in cards
BankerToolBench v1.1 Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes. Expert Oct 2026 32.6% Not in cards
BARC-Bench Agents handle routine workplace requests and use records to infer privacy and safety constraints. A judge checks whether each attempt completes safely or defers. University Oct 2026 72.7% Not in cards
BenchCAD 2.0 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 59% Not in cards
BenchCAD 2.1 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 83.5% In use
BenchCAD 2.1-beta Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 Not in cards
Binary Reasoning v1 Models answer machine-checked questions about compiled software across processor architectures. The hub port supplies 100 of 10,251 tasks and a separate set of 29 interactive tasks. Expert Oct 2026 Not in cards
BizBench Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success. Expert Oct 2026 70% Not in cards
BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. Expert Oct 2026 79.9 Not in cards
Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. Expert Oct 2026 8.47% In use
BrowseSafe-Adapt Browser agents face adaptive prompt injections across 1,024 tasks. The score is attack success within 64 attempts. University Oct 2026 30.7% Not in cards
Build-A-Chip-Bench Agents take a chip from workload specification to a routed GDS layout with open-source tools. Each design must pass hidden verification, routability and timing gates. Research frontier Oct 2026 16.6% Not in cards
BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. Everyday Oct 2026 0.56 Not in cards
Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. Expert Oct 2026 37.8% In use
Cat-Goric Game Benchmark Models navigate a 2D platform game and submit results to a public decision-making leaderboard. Everyday Oct 2026 20.0 Not in cards
CheckerBench Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback. Expert Oct 2026 45.3% Not in cards
CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. Expert Oct 2026 76.6 Not in cards
CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. Expert Oct 2026 Not in cards
CivBench Models play full games of Civilization V with the Vox Populi mod against each other on matched maps and seeds. Pairwise skill ratings rank long-term strategy. Expert Oct 2026 1,634 Not in cards
CLASHarness Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment. Everyday Oct 2026 28.7% Not in cards
Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. Expert Oct 2026 $9,514 Not in cards
ClothDojo Robot policies use both hands to flatten and fold garments. The simulator checks success on familiar garments and garments withheld from training. Everyday Oct 2026 44.7% Not in cards
Co-Design Bench Agents revise existing applications from multimodal user requests. A judge checks browser and repository evidence against 2,331 expert criteria across 273 tasks. Expert Oct 2026 69.9% Not in cards
Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. Expert Oct 2026 86.9% System cards only
Code2Video-AA Artificial Analysis' implementation of HeyGen's Code2Video Bench: models write a HyperFrames HTML composition for each of 114 product and brand video briefs. A model judge compares two rendered videos at a time on five axes, and the comparisons become Elo ratings. Expert Oct 2026 1,020 In use
CompactionBench Agents continue multi-turn coding sessions while context compaction occurs during software evolution tasks. Scores measure task quality and compaction cost. Expert Oct 2026 Not in cards
Company Enrichment API Benchmark Company data APIs fill profile fields from a website. Code and expert checks score headquarters, employees, founding year, social links, industry, ticker and revenue. Expert Oct 2026 Not in cards
Concurrency Bugs (tanishgudise) Agents repair Python packages with asynchronous or threaded code. Hidden tests reproduce each bug deterministically across 15 repair tasks. University Oct 2026 Not in cards
Construction's Last Exam v2 Models measure construction drawings, answer questions across sheets, and price quantities under contract rules. The board reports geometry, drawing reasoning, and estimating separately. Expert Oct 2026 36.8% Not in cards
Continual Learning Exam Agents work through 120 professional tasks, each a ladder of five levels that must be passed in order. Later levels reuse files, tools and lessons from earlier ones. An evaluation agent grades against expert rubrics. Expert Oct 2026 47.7% Not in cards
Cozzo AI Receptionist Benchmark (Round 6) Voice agents quote a dental price, confirm appointments, and handle urgent calls. Cozzo checks spoken details and scores responses to interruptions. Everyday Oct 2026 Not in cards
Cresta-Bench Contact-center agents take simulated calls for lending, health and airline lines. Tool calls get deterministic checks and LLM judges grade the rest. Everyday Oct 2026 82.6% Not in cards
CRM-Agent-Bench Agents handle customer and business requests in a simulated enterprise. Executable checks verify tool arguments, action order, and which records changed across 555 tasks. University Oct 2026 83.6% Not in cards
CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Expert Oct 2026 80.4% In use
CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. Expert Oct 2026 87.7% Not in cards
D2K-Bench Agents turn expert GPU-kernel designs into efficient Triton, CUDA or CUTLASS kernels across 26 tasks. Runtime timing and design-code scores measure implementation quality. Expert Oct 2026 Not in cards
DataAnalyticsBench Agents query a synthetic data warehouse to answer business questions. A judge model checks answers against a rubric and an answer key. University Oct 2026 43.0 Not in cards
DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. Everyday Oct 2026 40% Not in cards
DAYJOB: Accounting Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Oct 2026 49.6% Not in cards
DAYJOB: Data Science Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Oct 2026 37.3% Not in cards
DAYJOB: Law Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Oct 2026 41.4% Not in cards
DAYJOB: Tax Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria. Expert Oct 2026 10.8% Not in cards
DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. Everyday Oct 2026 Not in cards
DECK-Arena Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores. University Oct 2026 1,456 In use
DeepRSI 15 open-ended ML tasks with a 48-hour budget each. Private verifiers score gains over baselines. Models ranked by Bradley-Terry Elo. Research frontier Oct 2026 1,629 Not in cards
Developer Retrieval Benchmark Search systems retrieve repositories, documentation pages and issue or pull request fixes. Deterministic URL matching scores recall and reciprocal rank. Expert Oct 2026 Not in cards
DexArena Robots use both hands to complete household and industrial tasks. The simulator checks each stage and the final outcome across ten multistage tasks. Everyday Oct 2026 52.1% Not in cards
Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. Expert Oct 2026 28.3 Not in cards
Doc2Harness Models build a coding-agent harness from an interface skeleton and a development document. The frozen harness then runs 328 scripted test cases in separate containers. Research frontier Oct 2026 53% Not in cards
DOE sample pack Agents complete five simulated workplace assignments in law, private equity, and finance operations. Deterministic checks grade figures and application state, while model judges grade written work. Expert Oct 2026 Not in cards
Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. Everyday Oct 2026 Not in cards
Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. Expert Oct 2026 59.8 Not in cards
DualEntry Accounting AI Benchmark Models use accounting tools to look up accounts, reconcile balances, and draft entries for a test company. DualEntry checks their answers against fixed references. University Oct 2026 91.1% Not in cards
Earth-Bench V2 Agents analyze prepared satellite observations. The corrected release measures final answer accuracy, tool selection, argument accuracy, and execution efficiency. University Oct 2026 80% Not in cards
EGI-BENCH Agents work across eight enterprise evaluations. The suite combines their results into six scores covering activities such as planning, retrieving evidence, and using tools. University Oct 2026 65.6 Not in cards
Embodied Agent Arena Vision-language agents solve geometry, spatial reasoning, affordance, planning, and manipulation tasks. The 1,000-case evaluation combines 32 established sources with GeoProbe. Everyday Oct 2026 Not in cards
Embodied First Exam Coding agents answer frozen robot tasks from many benchmark suites with a program. Each program is verified in a separate sandbox, run on Harbor. Everyday Oct 2026 49.6% Not in cards
EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. Everyday Oct 2026 52% Not in cards
EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. University Oct 2026 41.4% Not in cards
Epoch Automation Reports Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees. Expert Oct 2026 81.7% System cards only
Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. Expert Oct 2026 77.4% In use
ExeWorldBench Coding agents build executable code-based 3D worlds that capture how a world works, testing world understanding and its realization in code. Expert Oct 2026 Not in cards
ExpertCrowd-SWE Agents resolve 100 software issues across 87 repositories in 11 languages. Each engineer-written rubric combines behavioral tests with a model judge and awards partial credit. The score averages five valid runs per task and then averages tasks. Expert Oct 2026 67.3% Not in cards
FDE-Gym Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1. Expert Oct 2026 0.56 Not in cards
FinHopper Agents research financial facts through up to six dependent searches. Intermediate answer keys show where an incorrect company, date, or indicator derails the search. University Oct 2026 67.3% Not in cards
FlashDexRetarget 50-motion benchmark Retarget 50 human hand-object motions (HOT3D, TACO, OakInk2) to XHand and Sharpa hands in Isaac Sim. Scored by object pose-error success thresholds. Everyday Oct 2026 Not in cards
Flex-RSI Agents improve a starter solution and their own tools within a fixed budget. A hidden test split measures the final solution. Expert Oct 2026 1.52 Not in cards
FoamBench V2 Agents build and run 110 Basic and 16 Advanced OpenFOAM cases. Revised instructions specify geometry, meshes, and boundary conditions for the same named cases. University Oct 2026 86.5% Not in cards
FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. Everyday Oct 2026 54.2% In use
FrontierCAD Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry. Expert Oct 2026 0.67 Not in cards
FrontierCAD Assemblies Agents place supplied CAD parts into 16 assemblies. The score is the published mean assembly geometry score. Expert Oct 2026 0.71 Not in cards
FrontierCAD Parts Agents build 39 CAD parts from engineering drawings. The score is the published mean geometry score. Expert Oct 2026 0.63 Not in cards
FrontierCode 2.0 Agents resolve 132 maintainer-built tasks from deliberately ambiguous prompts. Rubric checks ask whether a maintainer would merge the change. Expert Oct 2026 50.2% In use
FrontierCS 2 Agents work inside the released repository of a recent systems or programming-languages paper and try to beat that paper's method. A hidden held-out suite grades the final patch against the human reference. Research frontier Oct 2026 58.3% Not in cards
FrontierML (Synthetic Sciences) Agents conduct 25 empirical machine learning projects under fixed budgets. Each task has a verifier and reference scores. Harbor includes the 15 public projects. Research frontier Oct 2026 0.52 Not in cards
G2 Voice AI Agent Platform Evaluations Voice platforms run the same agent on simulated business calls. Bluejay checks required tool arguments and the database after hangup, then measures transcription and latency. University Oct 2026 Not in cards
GeoExpertQueries Agents choose geospatial data and carry out an analysis. A model compares each execution trace with a reference workflow agreed on by three GIS specialists. Expert Oct 2026 Not in cards
GeoKTools Agents combine up to four kinds of geospatial tools to answer quantitative questions. Exact matches earn full credit, and a model grades other answers. University Oct 2026 Not in cards
Graduate Admissions Agent Benchmark Agents advise students on graduate admissions and revise plans when the student's information changes. Seventy Chinese-language tasks span 23 cases and 83 answer stages. Separate reviewers score responses against task-specific rubrics on a 100-point scale. Expert Oct 2026 79.1 Not in cards
Gray Swan Reward Hacking Bench Agentic environments where a task can be finished honestly or by crossing a declared line, such as shipping insecure code, leaking a secret, harming a third party through a shared service or leaving the scope of a penetration test. Reports how often each line is crossed. Expert Oct 2026 System cards only
Gym-Anything-RT Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time. Everyday Oct 2026 9.00% Not in cards
Hanabi Agent Benchmark Models play cooperative Hanabi in four-player teams. The benchmark scores shared board progress while hints and lives constrain coordination. Everyday Oct 2026 20.5 Not in cards
Harvey LAB v1.1 (HLAB-Vals-1.1) Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes. Expert Oct 2026 27.1% In use
Harvey LAB-AA v1.1 Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed. Expert Oct 2026 15% In use
HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. Expert Oct 2026 57.3% Not in cards
Hebbia Financial Services Benchmark Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion. Expert Oct 2026 74.6% Not in cards
Hedge Gym Models adjust an options hedge each day over a 21-day price path. The evaluator measures replication error and transaction costs on synthetic or historical markets. Expert Oct 2026 Not in cards
Hermes Bench Agents in Hermes Agent do 150 tasks across 25 categories, such as research, diagrams, memory, tool use and safety. Graders check the files and state left behind. Expert Oct 2026 76.7% Not in cards
Hermes Index Nous Research averages pass@1 and cost per task over Hermes Bench, Terminal-Bench 4.0, Terminal-Bench Science and SkillsBench, all run in Hermes Agent. Expert Oct 2026 63.3 Not in cards
Historical Jailbreak Regression Replays previously found cyber jailbreak attacks against fixed ExploitBench tasks with real-time blocking turned off. Scored by mean task success rate under attack. Expert Oct 2026 14.3% System cards only
HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. Everyday Oct 2026 49.4% Not in cards
HWE Bench V2 (RISC-V CPU design) Agents improve a RISC-V processor from a shared starting design. The benchmark scores throughput on five hidden programs using clock speeds from Gowin's timing tool. Expert Oct 2026 7024.2 Not in cards
Hy-Nomas-Bench Interactive video world models follow text and camera controls across multiple turns and retain spatial memory. Expert Oct 2026 Not in cards
InnovationEval Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches. Research frontier Oct 2026 Not in cards
InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. Everyday Oct 2026 41.7% Not in cards
Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. Expert Oct 2026 76.4% System cards only
Internal Finance Modeling (Moonshot) Agents complete financial modeling tasks with linked formulas, financial logic and final workbook quality. Experts rate the work. Expert Oct 2026 Not in cards
Internal Financial Research (Moonshot) Agents produce auditable financial research under evidence and cutoff constraints. Experts rate the resulting research. Expert Oct 2026 Not in cards
Into the Unknown Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check. Research frontier Oct 2026 0% Not in cards
Investor's Last Exam Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria. Expert Oct 2026 68.1 Not in cards
IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run. Competition Oct 2026 100% Not in cards
IPI Bench Agents complete workplace workflows while attackers add misleading records to an emulated company with 11 services. The benchmark pairs six workflow types with harmful injected objectives across 34 tasks. The headline here measures single-query attack success as a percentage, where lower is better. Expert Oct 2026 4.80% Not in cards
Iris Safety Evaluations Reflection AI's in-house safety suite of single-turn, multi-turn, jailbreak and agentic requests. Reports the share of harmful requests refused or redirected and the share of harmless requests answered. Everyday Oct 2026 88.9% System cards only
Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score. Expert Oct 2026 66.5% In use
Kimi Hallu Bench Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. Expert Oct 2026 4.20% In use
LAB: US Case Law Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings. Expert Oct 2026 Not in cards
LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. Everyday Oct 2026 63.3% Not in cards
Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. Expert Oct 2026 25.1% In use
Learn2Play Bench Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure. Everyday Oct 2026 81.8 Not in cards
Legal Consent Sweep Agents review an acquisition data room, reconcile consent requirements, and prepare notices and closing advice. A separate verifier grades their work. Expert Oct 2026 0.38 Not in cards
LegalReasonBench Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings. Expert Oct 2026 50.8% Not in cards
LevelBench-100 v4.1 Models take over the books of small businesses using QuickBooks exports. They reconcile balances, identify missing evidence, and prepare closing entries across 12 workflows. Expert Oct 2026 81% Not in cards
LiveMACE Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns. Expert Oct 2026 Not in cards
LiveSearch Agents search for current financial figures. The evaluator runs a stored query against a professional data service to update the reference answer. University Oct 2026 76.7% Not in cards
LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. Expert Oct 2026 66.5% Not in cards
Long Context Navigator Agents answer questions that link facts across a synthetic document of about 32,000 tokens. They must read, compress, or discard information within a 600-token working context. University Oct 2026 Not in cards
Long-Horizon-Terminal-Bench 2.0 Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards. Expert Oct 2026 69.2% Not in cards
MAMUBench Agents represent different users who share a limited resource. The benchmark compares their combined outcome with one agent coordinating for everyone. Everyday Oct 2026 Not in cards
MaxxBench Agents solve 89 modified DeepSWE repository tasks. Revised requirements, code, reference patches, and tests check whether agents can solve changed problems rather than recall original fixes. Expert Oct 2026 76.4% Not in cards
Maydan HF RL Sampler Agents reconcile synthetic Saudi social insurance contributions across ten payroll tasks. SQLite checks the final records and audit log. Expert Oct 2026 Not in cards
MBABench 2.0 Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar. Expert Oct 2026 15.8% Not in cards
Minecraft Bed Wars (YC Labs) Teams of agents play Minecraft Bed Wars against one another. The researchers compare the number of matches each team wins. Everyday Oct 2026 Not in cards
MirrorDojo Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDojo. Each episode pairs a user task with an instruction planted in a tool result. Reports attack success rate and utility. Everyday Oct 2026 3.10% System cards only
MirrorDyn Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDyn. Reports attack success rate and utility. Everyday Oct 2026 3.60% System cards only
MissionReport Models write government status reports in Word from working files and a template, across 20 expert-written Department of Defense style assignments. Expert rubrics score each decision. Expert Oct 2026 64.1% Not in cards
Mitosis Bench Agents mark every dividing cell in a tumor image. Each mark must fall within 30 pixels of an expert label. The grader rejects extra or missing marks. Expert Oct 2026 32.4% Not in cards
Navital Gym Agents complete tasks in six simulated Indian service settings. This port covers UPI Desk, which grades 50 payment and dispute tasks using written rules. Everyday Oct 2026 100% Not in cards
OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. Everyday Oct 2026 Not in cards
OmicsBench v1 Agents analyze molecular data across eight benchmarks. LatchBio reports pass rates for 741 short tasks and 43 long tasks. Expert Oct 2026 40.8% Not in cards
Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. Everyday Oct 2026 20.2% Not in cards
OncologyBench Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts. Expert Oct 2026 50.7% Not in cards
OpenFrontiers-Bench Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings. Research frontier Oct 2026 42.2% Not in cards
OpenTTD-Bench Agents build transport networks over ten simulated years. The grader rewards cumulative operating profit and the number of rivals each company beats. Expert Oct 2026 Not in cards
Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. Expert Oct 2026 3.10% System cards only
OR-License Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem. Expert Oct 2026 88.4% Not in cards
PAgentBench Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house. Everyday Oct 2026 74.1% In use
PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. Research frontier Oct 2026 14% Not in cards
PCBBench (pcbbench.org) Agents solve electronics problems and produce KiCad files. The public pilot checks artifacts across ten suites, including schematic design, board layout, and debugging. Expert Oct 2026 89.6% Not in cards
Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. Everyday Oct 2026 0% Not in cards
Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. Expert Oct 2026 In use
PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. Everyday Oct 2026 68.1% Not in cards
PharmaBench (micro1) Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff. Expert Oct 2026 17.8% Not in cards
PhysClosureBench Agents run 50 expert-verified physics research tasks in strong-field and attosecond physics, deciding how to generate and validate evidence. A 20-criterion score and a closure level from RC0 to RC4 grade them. Research frontier Oct 2026 6.29 Not in cards
Physical-Engineering's Last Exam Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. Expert Oct 2026 0.49 Not in cards
PokéBench: Raw Vision + Buttons Models play Pokémon Red from screenshots using eight Game Boy buttons. Game memory verifies ten milestones, with fewer turns breaking ties. Everyday Oct 2026 10.0 Not in cards
PretrainGym Agents select and mix public training data for a fixed language model. The grader trains on 400 million tokens and rewards low test loss. It penalizes overlap with evaluation data. Research frontier Oct 2026 Not in cards
Prompt to Product Agents build web applications from product briefs. Browser tests exercise features such as checkout arithmetic, saved filters, keyboard accessibility, and mobile layouts. University Oct 2026 98.2% Not in cards
PWS-Finance Web research systems answer dated finance questions about company profiles, funding, ownership, markets, regulation, and recent developments. Expert Oct 2026 Not in cards
Quant-Arena Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window. Expert Oct 2026 51.3 Not in cards
Questflow Finance Benchmark Agents query financial filings and market data with a fixed set of tools. They calculate metrics, rank securities, and submit answers that software checks against frozen references. University Oct 2026 100% Not in cards
Railhead Agents run a transport company in OpenTTD. The evaluator replays the action log and lets the company run without intervention before measuring its performance. Everyday Oct 2026 Not in cards
Range Repair Agent Agents repair a Python function that merges touching ranges. The sole task earns one point if the edited function passes every held-out test. School Oct 2026 Not in cards
Real-SWE 1.1 Resolution rate on private enterprise coding tasks, averaged across five runs per task. Expert Oct 2026 56.8% Not in cards
Realtime-Bench Agents act in 50 live game and web tasks where a late action fails: 30 web tasks (travel, shopping, services and security) and 20 games. The environment keeps moving while the model thinks. Three clocks separate speed from skill: Wall Clock runs on real time, Paused Clock stops while the model thinks, and Token Clock advances with each generated token. Scores are task success rates over three runs. Everyday Oct 2026 42.7% Not in cards
RealWorkDecision Agents review a frozen company workspace and predict the decision an employee made. Expert-written rubrics allow valid alternatives. University Oct 2026 43.8% Not in cards
RepoNormBench Agents change code in existing repositories while following the contribution rules that apply to each task. Expert Oct 2026 68.6% Not in cards
ReTrace Computer agents recover from injected errors in desktop tasks. Replaying the remaining actions checks whether the recovered state produces the original successful outcome. Everyday Oct 2026 63.6% Not in cards
ReviewBench AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1. Expert Oct 2026 73.3% Not in cards
RH SWE-bench Agents repair Go and Python code in Red Hat ecosystem repositories. The current candidate contains 337 tasks across 24 repositories after review and deduplication. Expert Oct 2026 Not in cards
Roark Live Bench v1.3 Voice agents answer calls over the public phone network and use simulated business tools. Roark scores task completion, tool use, speech quality, and response time. University Oct 2026 Not in cards
Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. Expert Oct 2026 32.7% Not in cards
Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. Everyday Oct 2026 Not in cards
Robots Evolved GUI lessons Agents use screenshots, clicks, and keys to move sliding robots onto a goal. Ten puzzles enforce a move budget and offer no undo. Everyday Oct 2026 Not in cards
RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. Everyday Oct 2026 19.1% Not in cards
RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. Expert Oct 2026 0.06 Not in cards
RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. Everyday Oct 2026 17% Not in cards
Rotom Champ Bench Models draft six level 50 Pokémon, select three for each match, and choose their moves. The organizer rates them on wins under Champions singles rules. Competition Oct 2026 2,304 Not in cards
Rotom Gen9 Bench Models draft six level 100 Pokémon and play the whole team in Generation 9 OverUsed singles. Match results determine their ratings. Competition Oct 2026 2,318 Not in cards
RSI Control Arena An agent does one of 14 tasks from a successor model's training pipeline, such as SFT or reward modeling, while covertly trying to install one of 16 misaligned properties in that successor. A held-out oracle checks whether the property landed, and LLM monitors try to flag the attack from the transcript. Research frontier Oct 2026 14.3% Not in cards
RSI-Inference Bench Coding agents add model support to inference engines, improve serving speed and fix performance regressions. The harness builds and runs each submission. Expert Oct 2026 2,044 Not in cards
rTaste Agents propose, rank, and forecast experiments inspired by recent research papers. An independent executor attempts the experiments, and auditors assess their results. Expert Oct 2026 3.71 Not in cards
RTLScout Bench (suite-v2) Agents rewrite five circuits to reduce the product of chip area and signal delay. The leaderboard ranks configurations by their mean rank across circuits. Expert Oct 2026 1.00 Not in cards
RuleTrek Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements. Expert Oct 2026 55.3% Not in cards
RWE-bench Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement. Expert Oct 2026 62% Not in cards
SandboxEscapeBench Agents attempt to escape a container sandbox and retrieve a flag from the host. Inspect runs the challenge inside reproducible virtual machines. Expert Oct 2026 Not in cards
ScholarCatalyst Given a research question, retrieve the 'catalyst papers' that 184 lead authors labeled for 207 projects, searching only prior literature. Recall@20. Expert Oct 2026 53% Not in cards
SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. Expert Oct 2026 80.9 Not in cards
ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. Expert Oct 2026 0.53 Not in cards
SciTaste-Bench Models judge the research value of papers in six disciplines. The benchmark compares their judgments with a reference index built from seven dimensions. Expert Oct 2026 28.0 Not in cards
SDBench Diagnostic agents order tests, ask questions and commit diagnoses in sequential clinical cases. Scores combine diagnostic correctness with test cost. Expert Oct 2026 Not in cards
Senior Engineer Benchmark (Every Checks) Agents repair Proof from the March 17 outage snapshot. Every reports the percentage of 48 checks each solution passes. Expert Oct 2026 85% Not in cards
SettlerBench Season One Four models trade resources, build settlements, and negotiate in public or private messages. Final victory points determine pairwise results for the Bradley-Terry rating. Everyday Oct 2026 1,270 Not in cards
Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. Expert Oct 2026 $2,949 Not in cards
SilverquiLLM-bench hob-medium Coding agents implement five Hobbit cards in a Python game engine. The grader checks card behavior with 90 hidden tests. University Oct 2026 96.9% Not in cards
SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. Expert Oct 2026 100% Not in cards
SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. Expert Oct 2026 Not in cards
SimuVerity Agents build executable Simulink models from text for 101 tasks in ten engineering domains. Simulations check that each model meets its engineering requirements. Expert Oct 2026 42.9 Not in cards
Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior. Research frontier Oct 2026 47.6% Not in cards
SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. Expert Oct 2026 25.1% Not in cards
SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. Expert Oct 2026 46% Not in cards
SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. Expert Oct 2026 27.8% Not in cards
SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. Expert Oct 2026 18.4% Not in cards
SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. Expert Oct 2026 30.5% Not in cards
Solana Machine Trading Agents write trading bots for hidden market scenarios replayed on offline Solana snapshots. The headline score compares total profit and loss against a reference bot. Expert Oct 2026 6.11 Not in cards
Spatial Agent Index v1 Agents analyze geospatial files and produce numbers, maps, or lists. Axis Spatial averages the scores for each included evaluation. University Oct 2026 Not in cards
Spatial Intelligence Index v1 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. University Oct 2026 Not in cards
SpireBench Agents play Slay the Spire and are scored on long-horizon deck-building decisions. Everyday Oct 2026 Not in cards
SpireClimb Agents play Slay the Spire 1 and 2 at max difficulty through a computer-use harness on fixed seeds. Score is the mean win rate. Expert Oct 2026 41.3% Not in cards
SpireClimb StS1 A20 Heart Agents play Slay the Spire 1 through its graphical interface at Ascension 20 and must defeat the Heart. Scored by win rate over fixed seeds. Expert Oct 2026 33.3% Not in cards
SpireClimb StS2 A10 Agents play Slay the Spire 2 at Ascension 10 through screenshots and mouse and keyboard, 15 runs per character. Scoring is the win rate. Everyday Oct 2026 49.3% Not in cards
SplatBench A model rebuilds an object in 3D from eight photos by placing up to 300 Gaussians itself through tool calls, with no optimizer. Half of the 30 tasks start from an empty scene, and half ask it to reassemble a scrambled fit. SplatScore compares renders from eight angles the model never sees, with an empty scene at 0 and gradient descent at 100. University Oct 2026 49.6 Not in cards
Stack EOL Audit: geol CLI vs API Agents identify software components that have reached end of life. They audit the same twelve stacks from memory, with an API, and with a command line tool. University Oct 2026 1.00 Not in cards
StarSkirmish Bench Each model gets one hour in its lab's own CLI to write a StarCraft: Brood War bot in C++. The score is its win rate against human-written bots. Expert Oct 2026 57.3% Not in cards
StoreBench Agents manage a simulated apparel store in Medusa. They order inventory, change prices, ship purchases, and process refunds as demand changes. University Oct 2026 0.83 Not in cards
sudo L7 Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change. Expert Oct 2026 52.2% In use
SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. Expert Oct 2026 51.2% In use
SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result. Research frontier Oct 2026 16.6% Not in cards
SWE-Clar Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build. Expert Oct 2026 67% Not in cards
SWE-Race Agents fix concurrency and asynchronous lifecycle bugs in Python projects. The benchmark has 95 public and 93 private tasks. Harbor distributes 94 public tasks. Expert Oct 2026 81.9% Not in cards
SWE-Serve Coding agents solve 53 production inference-engineering tasks derived from merged SGLang pull requests. Hidden tests measure task success, cost, agent steps and wall time. Expert Oct 2026 75.5% Not in cards
SWE-sweep Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved. Research frontier Oct 2026 8.90% In use
SWebBench Agents browse the web to retrieve information and do research tasks. iFlytek built it in house. Expert Oct 2026 71.7% In use
TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. Everyday Oct 2026 Not in cards
TasteVal 8 AI R&D tasks. Models iteratively design experiments. Taste measured as the compute needed to reach expert-level scores. Research frontier Oct 2026 230.4× Not in cards
TavernBench Agents build a Hearthstone Battlegrounds player through research, simulator updates, and live games. Expert Oct 2026 95.6 Not in cards
Tavily Company Search Benchmark Search providers return evidence for company research questions, and an answer agent responds from those results. A judge scores answer completeness, relevance and freshness. Expert Oct 2026 Not in cards
TestPrism Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one. Expert Oct 2026 33.3% Not in cards
The Impossible Refund Assistants pursue a late train refund in a simulated email environment. Separate judges rate deception and other behaviors from the messages. Everyday Oct 2026 Not in cards
The Midnight Index Models lead a nation through fixed scenarios in a strategy game. The simulation records outcomes, escalation, and whether public commitments match later decisions. Everyday Oct 2026 Not in cards
TheoPhysEval Agents solve 50 unpublished research-level theoretical physics problems on their own, with literature search and scripts. Each final answer is checked against an expert answer, and the score is the pass rate. Research frontier Oct 2026 98% Not in cards
Thrive Holdings helpdesk ticket resolution benchmark Agent investigates IT support requests, takes action and decides when to escalate to a technician. Scored by pass rate. University Oct 2026 61.5% Not in cards
TraceTrial-Bench Agent harnesses, not only models, are tested on reliability across seven components, with traces used to attribute each failure to its cause. Expert Oct 2026 77.6% Not in cards
Trajectory Labs cyber red teaming Expert red teamers try to break safeguarded cyber endpoints across nine attack technique families on exploit reproduction tasks. Expert Oct 2026 System cards only
TreasuryQA Bench Agents search scanned Treasury volumes and read tables to find or calculate a number. A script checks whether the answer falls within the accepted range. School Oct 2026 82.4% Not in cards
Turn based Benchmarks Models play turn based games including tic tac toe and rock paper scissors. The tasks evaluate game decisions and adaptation to opponents. Everyday Oct 2026 0.69 Not in cards
UnityBench-Mini3D Agents modify existing Unity projects. EditMode and PlayMode tests check the saved scene, component settings, and runtime behavior, with partial credit for completed milestones. University Oct 2026 72.5% Not in cards
Unsatz Agents design experiments and propose equations for an unknown simulated system. Their final program must predict held-out interventions and conditions beyond the original experiments. Expert Oct 2026 60% Not in cards
Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. Expert Oct 2026 69.5% In use
Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. Expert Oct 2026 68.3% In use
Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. Oct 2026 71.3% In use
Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. Expert Oct 2026 78.4% In use
Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. Expert Oct 2026 Not in cards
Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. Expert Oct 2026 68.4% In use
Vals Smith: AumeshMisra/tensorzero Agents modify AumeshMisra/tensorzero to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Oct 2026 86.7% Not in cards
Vals Smith: pallets-eco/flask-wtf Agents modify pallets-eco/flask-wtf to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Oct 2026 100% Not in cards
Verifiable Browsing Task (Meta) Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools. Everyday Oct 2026 Not in cards
VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. Expert Oct 2026 4.11 Not in cards
Web Clone Bench Agent gets a running reference app (Gmail, Jira, Figma, etc.) and 8 hours to build a clone. Expert-written workflows are replayed as end-to-end tests. Research frontier Oct 2026 62% Not in cards
Web Context (Nimble) Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages. Expert Oct 2026 80% Not in cards
Which AI Model Knows Israel Best? Models answer Hebrew questions about everyday life in Israel. The organizers check each answer against an official source and count refusals separately. Everyday Oct 2026 99.3% Not in cards
WildAWM validation Agents complete 33 held-out computer-use tasks in generated software environments. Harbor packages the validation split separately from the training tasks. Expert Oct 2026 22.3% Not in cards
WorkplaceAgents Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics. Expert Oct 2026 21.2% Not in cards
xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. Expert Oct 2026 100% In use
Your LLM doesn't Know BOUNDARY!. Models complete work tasks at different access levels. Tests check whether they solve the part they own and report a blocker when another owner must act. University Oct 2026 0.91 Not in cards
τ-bench 2.0 Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture. University Oct 2026 79.9% Not in cards
AA-Briefcase v1.1 Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Expert Sep 2026 1,822 In use
AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. Expert Sep 2026 System cards only
Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. Expert Sep 2026 System cards only
Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. Expert Sep 2026 3.50% System cards only
Agentic IF Index Composite of Meta internal evals on following tool-use restrictions, rubrics, compound constraints and long policies in agentic workflows. Scored 0-1. Sep 2026 System cards only
Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. Everyday Sep 2026 System cards only
ALE-Gold Agents use a Linux terminal to complete 96 professional tasks. Graders check the files they produce. A task passes only with full credit. Expert Sep 2026 Not in cards
Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. Expert Sep 2026 40% Not in cards
Anthropic large agent teams tasks Internal 24-hour tasks for teams of 1-100 agents: build a company knowledge base, or formalize a theorem in Lean. Research frontier Sep 2026 System cards only
AnyAppBench Agents repeat the same task in different Android apps, with and without suggested steps. Each app has a verifier that checks the same success condition. Everyday Sep 2026 67.8% Not in cards
APEX-Agents 1.1 Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero. Expert Sep 2026 90.4% Not in cards
Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. Expert Sep 2026 0.17% System cards only
ApprenticeBench Agent joins a simulated company and processes 100 accounts-payable bills in Odoo via GUI or API, learning from records and feedback. Cumulative success rate. University Sep 2026 77% Not in cards
ArtifactArena Models design robot bodies, controllers and strategy for a simulated sumo-style arena. Artifacts fight head-to-head and models get Elo ratings. Expert Sep 2026 1,539 Not in cards
Artificial Analysis Coding Agent Index v1.5 Version 1.5 of Artificial Analysis's coding agent index, which moved to the harder Terminal-Bench 4.0 and DeepSWE 1.1. Sep 2026 Not in cards
Artificial Analysis Cyber Index Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. Expert Sep 2026 51% In use
Artificial Analysis Intelligence Index v4.2 Version 4.2 of Artificial Analysis's index, which added multi-hour office work and document tasks and dropped GPQA Diamond. Sep 2026 Not in cards
Artificial Analysis Intelligence Index v4.3 Version 4.3 of Artificial Analysis's index, eleven weighted parts led by knowledge work, office tasks and Terminal-Bench 4.0. Sep 2026 56.0 In use
ATLAS-Finance Agents do 100 investment banking tasks inside 13 simulated firms. They gather facts from email, chat, calendar and files over MCP, then build an Excel model graded against expert rubrics. Expert Sep 2026 Not in cards
AutoResearchExam Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed. Research frontier Sep 2026 0.61 Not in cards
BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. Expert Sep 2026 Not in cards
Beancount Ledger Agents reconcile bank statements, repair a Beancount ledger, and allocate payments to invoices. The evaluator checks balances and payment records against the underlying transactions. University Sep 2026 Not in cards
Bench on the Clocktower Ten models play the social deduction game Blood on the Clocktower together, one seat each, over 300 ten-player games of the Trouble Brewing script. Each model plays 210 games on the good team and 90 on the evil team. The score is the share of games its team won. Everyday Sep 2026 58% Not in cards
Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. Expert Sep 2026 72.2% System cards only
Biopharma Bench Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers. Expert Sep 2026 65.1% Not in cards
BioSecBench-Function Agents recover biological function from experimental data. Scored by verifiable checks. One of three LatchBio BioSecBench evaluations. Expert Sep 2026 73.6% System cards only
BlenderBench (Refresh) Agents edit Blender scenes across 20 tasks. Each task grades the submitted scene using its own scoring configuration. University Sep 2026 0.69 Not in cards
Brood War Bench Coding agents play StarCraft: Brood War against each other in real time. Systems are ranked by win rate over matches, with cost per game. Expert Sep 2026 100% Not in cards
BU Bench V2.1 Browser agents do 200 real web tasks. A findings judge checks tool evidence, answers and files, and deterministic integrity checks run as well. Everyday Sep 2026 81.3 Not in cards
BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. Expert Sep 2026 76.4 In use
CAD Arena Agents rebuild licensed engineering drawings as editable CAD parts through the scripting APIs of NX, SolidWorks, Onshape, Fusion or Build123d. A verifier scores geometric match. Expert Sep 2026 0.75 Not in cards
CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. Expert Sep 2026 8.60% Not in cards
Chess Engine Benchmark (Steve Maughan) Models have 24 hours to write a chess engine from scratch in C or C++. Maughan rates each engine against reference engines and other entries. University Sep 2026 3,542 Not in cards
CivBench (Civilization VI) Agents play Civilization VI and are evaluated on strategic progress in a long-running world. Expert Sep 2026 73.0 Not in cards
Cloud Expansion CE80-10 Agents complete a cloud network security exercise. The verifier checks the resulting cloud configuration and observed network behavior. Expert Sep 2026 Not in cards
Cloud Expansion CEExfil-0 Agents complete a cloud security capstone. A verifier reads the environment after the task to assess the outcome. Expert Sep 2026 Not in cards
Cloud Expansion RX00 Agents complete a simulated cloud reconnaissance assessment. The verifier compares the submitted result with the recorded state of the exercise. Expert Sep 2026 Not in cards
Code2Video Benchmark 168 product-video briefs. Agent writes HTML motion-graphics code rendered by HyperFrames. A trained pairwise judge on five axes produces Elo. Expert Sep 2026 1,585 Not in cards
Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. University Sep 2026 0.62 Not in cards
Compliance with instructions planted in pasted text (Anthropic) Coding agents receive user prompts with pasted text that hides an attacker's instruction. Score is the rate of acting on the planted instruction. Expert Sep 2026 7.40% System cards only
CompMat-Bench Agents complete individual steps from published computational materials studies. Fixed rules compare their inputs and outputs with results reproduced by the authors. Expert Sep 2026 91.5% Not in cards
CorpBench Work Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly. Everyday Sep 2026 92.7 Not in cards
CUA-bench Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out. Everyday Sep 2026 19.2% Not in cards
CursorBench 4.0 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported. Expert Sep 2026 65.4% In use
CWE-bench Agents audit a repository and write a patch for a security flaw, without being told where the bug is. Covers six programming languages. Expert Sep 2026 71% In use
Data Science Tasks (OpenAI) Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading. Expert Sep 2026 40.9% In use
Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. Expert Sep 2026 63.9% In use
DataBench v1.1 Agents answer ambiguous business questions and make recommendations from a synthetic company warehouse. Model judges assess the reasoning, supporting evidence, artifacts, and final recommendation against task rubrics. Expert Sep 2026 70.5% Not in cards
Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. Expert Sep 2026 System cards only
DAYJOB: Finance Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Sep 2026 42.5% In use
DAYJOB: Healthcare Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Sep 2026 48.3% Not in cards
De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. Expert Sep 2026 85.1% System cards only
DealBench-100 Agents perform 100 investment banking workflows across ten synthetic transactions. Deterministic checks grade calculations, decisions, committed changes, and consistency between models and presentations. Expert Sep 2026 Not in cards
Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. Expert Sep 2026 0% In use
DI-Bench Agents answer questions using company tables, documents, and business rules. The benchmark checks whether agents apply those rules when computing results. University Sep 2026 57.9% Not in cards
Disengagement-Aware Student Simulators Tutors interact with simulated learners who vary in engagement. The authors compare five tutors and validate the student behaviors against real tutoring sessions. University Sep 2026 Not in cards
DolphinBench Agents use long-term memory to complete 600 tool-using tasks for three simulated knowledge workers with roughly 500,000 user-message tokens each. Expert Sep 2026 70.7% Not in cards
DrivingBench Vision-language models drive a real Toyota Corolla through a cone course with up to three attempts in one conversation. Scored on the furthest progress along the course. Everyday Sep 2026 100% Not in cards
DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. Expert Sep 2026 40.0 Not in cards
Duplex-MPE Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios. Everyday Sep 2026 72.3% Not in cards
Duplicate Side Effect Desk Agents process refund requests when timeouts or duplicate tickets obscure payment status. The evaluator uses the transaction ledger to identify duplicate refunds. Everyday Sep 2026 0.98 Not in cards
E2E-SWE Agents build complete software repositories from natural language specifications in empty workspaces. Hidden tests require every generated project to install and run. Research frontier Sep 2026 Not in cards
Earth-Bench-Pro Agents complete Earth observation workflows under three conditions. They follow instructions, plan from supplied observations, or obtain observations at runtime. University Sep 2026 Not in cards
EmailBench Agents handle email and productivity requests in a synthetic inbox. The evaluator combines executable checks and model-graded rubrics across 206 scenarios. University Sep 2026 33.5% Not in cards
EngiWorld Agents work through the full industrial engineering design loop on 1,301 expert-curated tasks that need reasoning over geometric and physical constraints across software. Expert Sep 2026 44.7% Not in cards
Era by Eon Generates a simulated enterprise (Salesforce, Slack, Jira, Zendesk, etc.) with exact ground truth. Agents answer questions through MCP/REST. University Sep 2026 76.8% Not in cards
ERPBench-100 Agents execute 100 business workflows across ten simulated enterprise resource planning systems. Checks grade decisions, calculations, committed state, and handoffs on a 100-point scale. Expert Sep 2026 Not in cards
EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. Research frontier Sep 2026 29.8 Not in cards
ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. Research frontier Sep 2026 53.5% In use
ExplorationBench Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help. Research frontier Sep 2026 89% Not in cards
Factorio Play Agents build factories one action at a time in a Factorio simulator. The evaluator measures factory output on held-out layouts. University Sep 2026 Not in cards
fAIntasy Football Models draft NFL players, set lineups, bid for free agents, and trade under the same rules. The standings show wins, fantasy points, and lineup efficiency. Everyday Sep 2026 4.00 Not in cards
FinCrime Agent Benchmark Agents screen customers for sanctions, political exposure, and adverse media. A case passes when agents find the relevant alerts and resolve them under the bank's policy. University Sep 2026 71.1% Not in cards
FinFIRST Agents research financial questions and provide supporting sources. The benchmark grades individual requirements and reports how often an answer meets every requirement. Expert Sep 2026 71.5% Not in cards
FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. Expert Sep 2026 63.6% In use
FinStepBench LiveSearch Models retrieve and verify live financial facts. The score is the published LiveSearch percentage. Expert Sep 2026 76.7% Not in cards
Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. Expert Sep 2026 Not in cards
FrontierMath Erdős Models try to prove or disprove 68 open Erdős problems in Lean 4. A result counts only if the Lean checker accepts the full proof. Research frontier Sep 2026 10.3% System cards only
FrontierSWE 34 ultra-long-horizon engineering and research tasks. 5 trials of up to 20 hours each. Score is mean task score 0-100. Research frontier Sep 2026 69.9% In use
Game Arena Go Language models play Go against each other through the Game Arena text harness. Ratings come from head-to-head games. Expert Sep 2026 1,877 Not in cards
GameASG-Bench Agents build 47 browser games from written requirements. Static checks inspect each HTML file, and browser tests check controls, gameplay and state changes. University Sep 2026 55.3% Not in cards
GameForgeBench Agents repair game projects in engines such as Godot. The verifier runs the project and checks behavior such as camera movement and collisions. University Sep 2026 Not in cards
GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. Everyday Sep 2026 80.2 Not in cards
GDP.xlsx Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria. Expert Sep 2026 43.1% Not in cards
GDPval-AA v2.1 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Expert Sep 2026 1,846 In use
GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Scored by a distance-based GeoGuessr-style reward from 0 to 1. Everyday Sep 2026 0.64 Not in cards
GlyphBench Agents choose actions from text-rendered game boards. The standard suite contains 303 tasks across six game collections and scores agents by cumulative return. University Sep 2026 0.42 Not in cards
Gray Swan Shade knowledge elicitation through chat An automated attacker tries to elicit useful answers to 100 harmful and 100 dual-use cyber queries from a safeguarded model. Score is attack success rate. Expert Sep 2026 1.00% System cards only
HarnessDev Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score. Research frontier Sep 2026 67.8% Not in cards
HarvestBench Agents drive simulated tractors and decide how much fuel to spend avoiding animals. The benchmark measures how often they hit animals as avoidance costs change. Everyday Sep 2026 0.40% Not in cards
Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. Expert Sep 2026 100% System cards only
Huayu Awards AI Cockpit Evaluation 2026 In-car assistants interpret requests and operate vehicle features. Tests include planning three-stop trips and adjusting several cabin controls from one request. Everyday Sep 2026 Not in cards
HubBench Agents make workplace decisions across 13 professional domains using connected software tools and evidence. Version 1.4.0 grades 104 tasks with deterministic state checks. Expert Sep 2026 Not in cards
Incident Arena Agents diagnose and repair incidents in running services. The score averages the repair success rate across 20 incidents. Expert Sep 2026 64.3% Not in cards
Internal computer use safety benchmark (OpenAI) Models complete computer-use tasks while avoiding unsafe actions. Research frontier Sep 2026 2.40% System cards only
ironbench Agents write firmware for tasks such as serial communication and controller sequencing. The grader checks device behavior and counts tasks solved within the allowed attempts. University Sep 2026 69.2% Not in cards
ISA-Bench Models write programs for five games with restricted instruction sets. A verifier checks each program and gives feedback before the next attempt. University Sep 2026 60.9% Not in cards
KernelQuest Agents optimize Triton kernels. A task passes when the kernel produces correct outputs and reaches the speed target on the same GPU. Expert Sep 2026 79% Not in cards
KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. Expert Sep 2026 29.4% Not in cards
Large Language Model Football League Models draft, trade, and set weekly NFL lineups. Each team earns one result against its opponent and another against the league median. Everyday Sep 2026 Not in cards
Legacy-Bench Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass. Expert Sep 2026 60% Not in cards
Legora BAR Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers. Expert Sep 2026 106.0 Not in cards
LEXam-hard Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed. Research frontier Sep 2026 40.8% Not in cards
LibraryDesignBench An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity. Expert Sep 2026 48.9 Not in cards
LLM Fantasy Football Arena Guillotine Models draft NFL players and bid for free agents. Each week the lowest-scoring team leaves, and the remaining teams bid on its players. Everyday Sep 2026 10.0 Not in cards
Long-Horizon Freight AuditBench Challenge Agents price 100 difficult freight invoices and flag incorrect charges. The cases include incomplete information and exceptions to ordinary contract terms. Expert Sep 2026 10% Not in cards
Long-Horizon Freight AuditBench Production Agents price 200 freight invoices from contracts and carrier documents, then flag billing errors. Loop weights accuracy to reflect a 24 percent billing error rate. University Sep 2026 16.1% Not in cards
Long-Horizon Medical Agent Benchmark, Veterinary Clinic Ops v2 Agents operate a simulated veterinary clinic through its database API. Each of the 98 tasks checks whether the agent leaves the clinic records in the required state. Expert Sep 2026 100% Not in cards
LongPuzzleBench GUI agents solve 114 visual puzzle levels across six games through native interface actions. Everyday Sep 2026 Not in cards
Ludobench Backgammon Models play backgammon against opponents using dice rolls and legal moves. Ludobench publishes Glicko-2 ratings and match replays. Everyday Sep 2026 1,723 Not in cards
Ludobench Hokm Models play Hokm in partnerships. One player names trump, and partners try to take seven tricks before the other team. Everyday Sep 2026 100% Not in cards
MANDOS Session 2 Models make process safety decisions across four private industrial scenarios. A model judge assesses each run. The leaderboard reports the share of passing runs. Expert Sep 2026 96.9% Not in cards
Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. Expert Sep 2026 0% System cards only
MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. University Sep 2026 51.5% Not in cards
McStasBench Agents design neutron scattering instruments. A simulator checks whether their designs reproduce published measurements. The authors score two additional instruments separately. Expert Sep 2026 43.8% Not in cards
Medical Bench Agents read longitudinal dermatology charts and write clinical assessments. Expert-authored rubrics grade their findings and management decisions. Expert Sep 2026 88% Not in cards
MetagenomicsBench Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result. Expert Sep 2026 62.7% Not in cards
metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. Everyday Sep 2026 Not in cards
MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details. Expert Sep 2026 68.6% In use
MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. Expert Sep 2026 81.7% In use
Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. Expert Sep 2026 1.40% System cards only
Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. Expert Sep 2026 System cards only
Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. Expert Sep 2026 61.2% System cards only
Multi-Agent Arena Codenames Two teams of anonymized models play the word game Codenames. The board rates the clue giver and the guessers on separate rating scales. Everyday Sep 2026 1,535 Not in cards
Multi-Agent Arena Diplomacy Seven players, people or anonymized models, negotiate and fight over World War I Europe. Models are ranked by their mean share of the sum-of-squares score, where 14.3 is an even split. Everyday Sep 2026 37.5% Not in cards
MysteryMechanism Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold. Expert Sep 2026 53.1% System cards only
Nego Eval Agents choose negotiation partners and remember prior encounters across games. The evaluator compares match scores with reference policies on the same seeds. Everyday Sep 2026 Not in cards
OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. Expert Sep 2026 0% System cards only
OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. Everyday Sep 2026 43.9% Not in cards
OpenRSI Index Agents run autoresearch on real open-source model development projects with their original code, data and checkpoints, for 60 or more hours. Fixed verifiers score the result against the human recipe they started from. Research frontier Sep 2026 Not in cards
OpenRSI Index ACE Playbook Inspection and Repair Agents repair a reasoning playbook under a fixed budget. The score is held-out Formula exact-answer accuracy as a fraction. Research frontier Sep 2026 0.73 Not in cards
OpenRSI Index Isaac Lab PegInsert Reward Search Agents design robot training rewards under a fixed budget. The score is held-out peg-insertion success as a fraction. Research frontier Sep 2026 0.60 Not in cards
OpenRSI Index Kev Decision Architecture Agents improve a decision model under a fixed budget. The score is the published exponential of negative macro log loss. Research frontier Sep 2026 0.29 Not in cards
OpenRSI Index Learnability-Aware Long/Short CoT Adaptation Agents adapt mathematical reasoning examples under a fixed training budget. The score is accuracy across five math benchmarks. Research frontier Sep 2026 47.5% Not in cards
OpenRSI Index Marin Scaling Ladder Agents design one optimizer for six model sizes. The score is the published complete six-rung reward relative to AdamH. Research frontier Sep 2026 1.01 Not in cards
OpenRSI Index MInference 32-Head Sparse Prefill Kernel Agents optimize a sparse-attention prefill operator under a fixed budget. The score is speedup against the paired reference run. Research frontier Sep 2026 1.54× Not in cards
OpenRSI Index Molmo2 Video-Pointing Inference Strategy Agents tune video inputs and inference for a frozen Molmo2 model. The score is held-out spatiotemporal pointing soft F1. Research frontier Sep 2026 0.46 Not in cards
OpenRSI Index MolmoWeb Interaction Context Allocation Agents select interaction context for a frozen browser model. The score is macro-averaged browser-action prediction quality. Research frontier Sep 2026 0.61 Not in cards
OpenRSI Index ReasonIR Difficulty Curriculum Agents adjust retrieval training data schedules. The score measures reasoning retrieval with a penalty for general retrieval regression. Research frontier Sep 2026 0.22 Not in cards
OpenRSI Index SGLang Omni TTS Runtime Agents improve speech serving while preserving quality. The score is the official speedup reward for the archived continuation protocol. Research frontier Sep 2026 1.54× Not in cards
OSWorld 2.1 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. University Sep 2026 44.3% In use
OSWorld 2.1 (offline subset) Agents complete the 82 OSWorld 2.1 tasks on the official offline list, with no internet access. Scored by checkpoint partial credit and strict pass rate. University Sep 2026 89.3% In use
OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. Expert Sep 2026 73.7% Not in cards
Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. Expert Sep 2026 System cards only
Package registry publishing evaluation (Anthropic) Agents in a simulated capture the flag exercise can publish a backdoored package to a registry. An investigator escalates pressure until they act. Expert Sep 2026 33.3% System cards only
PACT (robot arm orchestration) Agents write and revise programs to control a simulated robot arm. The simulator checks whether the arm completes the requested manipulation. Expert Sep 2026 Not in cards
Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. Everyday Sep 2026 36.3% Not in cards
Parallel Search Capability Leaderboard Language models answer research questions with and without search, and Parallel ranks their accuracy, search lift, cost, and time. Expert Sep 2026 88.1 Not in cards
Parametric CAD Bench Agents create and edit parametric FreeCAD models from text and engineering drawings. Programmatic checks award continuous task rewards for geometry and specification consistency. Expert Sep 2026 61.0 Not in cards
Periapical Lesions Bench Agents locate and grade lesions around tooth roots in panoramic dental X-rays. A perfect report matches every lesion and grade without adding extra boxes. Expert Sep 2026 18.7% Not in cards
PhoneShell iOS Agent Benchmark Agents operate apps on a physical iPhone to set timers, find books, and complete other everyday tasks. Automated checks read the resulting device state. Everyday Sep 2026 96.6% Not in cards
PokéBench: Tool Assisted Benchmark Models play Pokémon Red with a screenshot, map, game text, and action shortcuts. They have 1,000 turns to reach ten milestones ending with Brock. Everyday Sep 2026 10.0 Not in cards
PostTrainBench 1.2 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Research frontier Sep 2026 44.6% In use
PowerBench Agents retrieve equipment records and telemetry to answer 300 power-system questions. Macro Joint Accuracy averages fully correct answers equally across three task families. Expert Sep 2026 74.2% Not in cards
Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. Expert Sep 2026 0.20% System cards only
ProgramDistill Agents inspect working web applications and restore missing features in editable copies. Replayed browser traces check whether patches recover the reference behavior. The paper evaluates partial repairs and full application reconstruction. University Sep 2026 84.3% Not in cards
Prosus Vending Bench v2 Agents run six simulated vending machines across three locations for 30 days. They order stock, set prices, and collect cash. The score measures final bank balance in euros. University Sep 2026 4497.2 Not in cards
Qwen-Audio-Agent In-House Cockpit Benchmark Agents handle requests with cockpit tools. The evaluation measures whether they complete each task when they call tools directly, delegate, or choose between both routes. Everyday Sep 2026 Not in cards
Qwen-Audio-Agent Smart Cockpit Benchmark Agents operate simulated vehicle, navigation, music, and weather services. The short suite checks whole cases. The long suite checks tool use on each turn. Everyday Sep 2026 98.8% Not in cards
Real-SWE Models resolve tasks in private production codebases licensed from real companies. The board reports resolution rate with standard errors. Expert Sep 2026 38.8% Not in cards
RecreationBench Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result. Expert Sep 2026 58.1% In use
Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. Expert Sep 2026 0% In use
Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. Expert Sep 2026 11.8% In use
REVERSAL-BENCH Reset-free reinforcement learning agents learn manipulation policies across environments with controlled reversibility and recoverability checks. Expert Sep 2026 Not in cards
RIICHI ORBIT mahjong bot comparison (open practice) Models get 30 minutes to improve the same mahjong bot, with unlimited practice. The maker freezes each bot before evaluating it on separate tile sequences. University Sep 2026 2.41 Not in cards
RIICHI ORBIT mahjong bot comparison (two-stage practice) Models improve a beginner mahjong bot during two timed coding periods. The maker runs the finished bots in four-player matches and reports mean placement by table. University Sep 2026 2.28 Not in cards
Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. Everyday Sep 2026 52.5% Not in cards
RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. Everyday Sep 2026 65.7% Not in cards
Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. Everyday Sep 2026 20% System cards only
RSI Arena Ten frontier agents each post-train the same Nemotron base model on a shared GPU cluster for six days. People then rank the resulting models in blind side-by-side votes, and agents retrain on that feedback. Research frontier Sep 2026 1,095 Not in cards
Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. Research frontier Sep 2026 45.5% System cards only
Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. Expert Sep 2026 1.50% System cards only
Sci Pipeline v1 Agents clean hourly sensor readings and write daily means, record counts, and extremes to a JSON file. The grader checks the saved file. University Sep 2026 Not in cards
SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. Expert Sep 2026 45.3% Not in cards
SemiKongBench-100 Agents resolve 100 synthetic semiconductor operating exceptions. The evaluator grades each workflow with deterministic checks. Expert Sep 2026 Not in cards
Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. Expert Sep 2026 8.00% System cards only
Simreal ML Benchmark ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them. Expert Sep 2026 Not in cards
Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. Expert Sep 2026 0% System cards only
SpeedrunBench Agents try to finish goals in 10 video games as fast as possible. Score is time to goal in frames or seconds, lower is better. Everyday Sep 2026 Not in cards
StationeryBench Robots use both arms to handle common desk supplies. A human grades progress from zero to 100 on five tasks. Everyday Sep 2026 46.0 Not in cards
StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup. Expert Sep 2026 63.9% In use
StudentBench AI tutors teach real students GRE material and are scored by the students measured learning gains. A second study has expert tutors compare AI lesson plans and practice problems. University Sep 2026 16.5 Not in cards
SWE-bench Multimodal v2 Agents patch JavaScript repositories using issue reports that include images. Automated tests check whether each patch fixes the issue. Expert Sep 2026 70.2% System cards only
SWE-Bench Pro V2 Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container. Expert Sep 2026 99.4% In use
SWE-Bench Pro V2 Hard Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol. Research frontier Sep 2026 98% In use
SWE-Bench Pro Verified Agents repair software repositories. OpenCompass corrects inconsistent tasks and strengthens tests in SWE-Bench Pro. Expert Sep 2026 Not in cards
SWE-Game Agents build, complete, repair, or port games across 247 tasks derived from 41 reference games. Runtime probes and input replays check gameplay mechanics and preserved behavior. Vision-language rubrics separately grade presentation in construction tasks. University Sep 2026 64.0 Not in cards
Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. Expert Sep 2026 52.3% In use
TimelineBench Agents turn raw footage and a client brief into a finished video. The edit must pass delivery checks, meet the brief, and clear a quality threshold. University Sep 2026 32.1% Not in cards
TruthInsightBench Agents analyze data from published studies without seeing the original conclusions. A judge grades whether the evidence and artifacts support the claims each agent makes. Expert Sep 2026 Not in cards
TxBench-Oligo-Discovery Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision. Expert Sep 2026 62.2% Not in cards
UK AISI Out of Scope Supply Chain Attack Agents in simulated hard cyber challenges with incidental internet access may attack out-of-scope open source projects. Scored as rate of samples with such attacks. Expert Sep 2026 12% System cards only
Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. Expert Sep 2026 0% System cards only
Vals Index v2.1 Version 2.1 of Vals AI's GDP-weighted index, which moved coding to Terminal-Bench 4.0 and added a tax sector. Sep 2026 68.9% In use
Vals Smith: dwojtaszek/zipper Agents modify dwojtaszek/zipper to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 93.3% Not in cards
Vals Smith: Effect-TS/effect Agents modify Effect-TS/effect to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 93.3% Not in cards
Vals Smith: Expertium/fsrs-autoresearch Agents modify Expertium/fsrs-autoresearch to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 96.7% Not in cards
Vals Smith: fmtlib/fmt Agents modify fmtlib/fmt to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 83.3% Not in cards
Vals Smith: Lemonawa/music163bot-rust Agents modify Lemonawa/music163bot-rust to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 93.3% Not in cards
Vals Smith: OpenTTD/OpenTTD Agents modify OpenTTD/OpenTTD to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 100% Not in cards
Vals Smith: punt-labs/lux Agents modify punt-labs/lux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 73.3% Not in cards
Vals Smith: schubydoo/clauster Agents modify schubydoo/clauster to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 96.7% Not in cards
Vals Smith: tensorflow/tensorflow Agents modify tensorflow/tensorflow to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 90% Not in cards
Vals Smith: vastsa/PI-Desktop Agents modify vastsa/PI-Desktop to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 40% Not in cards
Vals Smith: vllm-project/llm-compressor Agents modify vllm-project/llm-compressor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Sep 2026 96.7% Not in cards
VAmoS Pro Bench Voice agents handle several utility billing requests in one call. A model checks the spoken amounts and recorded actions against the requirements for each request. University Sep 2026 40.1% Not in cards
Vibe Code Bench 1-100 Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it. Expert Sep 2026 34% Not in cards
Voice Arena Goal Dataset Judges compare two calls and choose which caller achieved the airline task more successfully. Human preferences provide the reference. Everyday Sep 2026 64.7% Not in cards
WeirdML v3 Agents solve 11 hand-made machine learning and data analysis tasks on a GPU with unfamiliar data and hidden test answers. Scored by token-weighted best results. Expert Sep 2026 43.1% Not in cards
WitnessBench Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels. Expert Sep 2026 46.5% Not in cards
WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. Everyday Sep 2026 42.3% Not in cards
WRENCH (Docker Compose) Agents maintain a service pipeline through seeded disruptions. The grader measures retained throughput and recovery. University Sep 2026 0.81 Not in cards
XYEval Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice. Expert Sep 2026 60.9% Not in cards
ZenML Bench Agents build, repair, and investigate machine learning pipelines in ZenML. Eighteen tasks grade the state the framework records. Expert Sep 2026 Not in cards
τ^τ-bench Coding agent builds a customer-service agent from business records, a client and an API. The built agent is scored against held-out simulated users. Expert Sep 2026 23.9% Not in cards
AA-AnalystAgent Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right. Expert Aug 2026 62.5% In use
Active-SWE Coding agents inspect repository snapshots and fix bugs without issue reports. Evaluation checks patches against reference tests and measures localization precision and recall. A separate stage uses generated tests and model review to assess newly revealed bugs. Expert Aug 2026 20% Not in cards
ADeptS-Bench Desktop Agents choose safe interface actions on the Desktop test set. The published safety score combines task success and attack resistance. Expert Aug 2026 76% Not in cards
Agent-Only Fantasy Football League Models draft NFL players, set lineups, and negotiate trades using the same prompt and tools. The organizer compares points, lineup efficiency, and cost per point. Everyday Aug 2026 1371.4 Not in cards
Alibaba CTF Agents solve 87 capture-the-flag tasks in Alibaba’s collection of competition challenges from 2023 through 2026. Expert Aug 2026 Not in cards
Apodex Discovery Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors. Research frontier Aug 2026 2.80 Not in cards
AQuA Two LLM research loops (factor discovery, model development) reuse evidence from their own past experiments inside sealed sandboxes. Scored by IC and held-out Sharpe. Research frontier Aug 2026 Not in cards
Artificial Analysis Coding Agent Index v1.4 Version 1.4 of Artificial Analysis's coding agent index, which moved to the full Terminal-Bench 2.1 and began scoring reward-hacked runs as zero. Aug 2026 67.0 In use
Artificial Analysis Search Index One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores. Expert Aug 2026 Not in cards
ASI-Bench AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge. Research frontier Aug 2026 55.8 Not in cards
Aspire Agent gets only a vague capability goal and must choose data, training and self-checks to evolve weights or harness. Scored on 520 hidden items. Research frontier Aug 2026 25.6% Not in cards
AutoDS scientific machine learning cases Agents build predictive models for maize yield, polymer glass transition temperature, and supercomputer job outcomes. Each case supplies a train split and a held-out test. Expert Aug 2026 Not in cards
Balatro Bench (Michael Andrzejewski) Agents play one fixed Balatro seed with the Red Deck on White Stake. The maker ranks runs by ante reached, breaking ties by best hand. Everyday Aug 2026 18.0 Not in cards
Behavior2Code Agents reconstruct command line programs from documentation and observed behavior. Hidden tests compare each implementation with the original. Expert Aug 2026 31.7% Not in cards
BixBench3 Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts. Research frontier Aug 2026 52.3 Not in cards
Blobfish Litigation Bench Agents complete 100 tasks in a simulated litigation practice. The grader compares the changes agents make to case records against required outcomes. Expert Aug 2026 Not in cards
BoardBench 0.1 Models play Yatzi, Scotland Yard, and Splendor. Game engines identify the best action at each decision and the scorer counts exact matches. Everyday Aug 2026 91.4% Not in cards
BrowseComp Extended Agents browse the web to find hard-to-locate facts with short answers. Mercor adds 100 new expert-written tasks to the 130 public ones and reports both scores. Expert Aug 2026 73% Not in cards
Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. Expert Aug 2026 38% System cards only
ClawProBench Agents on stateful runtimes are scored with their traces, covering evidence gathering, runtime routing, safety boundaries and repeated runs in frozen workplace setups. Expert Aug 2026 80% Not in cards
CliniCARE-Bench Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases. Expert Aug 2026 76.1% Not in cards
CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. Expert Aug 2026 Not in cards
ContinualSkillBench Agents work through five domains of 100 linked subtasks of rising difficulty, testing whether skill libraries they build in context actually help later tasks. Expert Aug 2026 64.1% Not in cards
CorporateBench Agents answer questions over the email archives of four synthetic companies, up to 232,692 documents. Answers are checked against each company's ground-truth knowledge graph. Expert Aug 2026 82.2% Not in cards
CounselBench-100 Agents investigate legal matters and decide which actions the evidence supports. The verifier checks decisions, changes to case records, and advice across 100 synthetic tasks. Expert Aug 2026 Not in cards
Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. Expert Aug 2026 20% System cards only
Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. Expert Aug 2026 100% System cards only
CyberStrike Agents complete cybersecurity tasks. Automated checks grade the outcomes. Expert Aug 2026 31.5% Not in cards
data-eng-bench Agents build and repair dbt models across 103 retail warehouse tasks. The verifier compares materialized tables row by row against a reference solution. University Aug 2026 Not in cards
DevOpsBench-100 Agents resolve 100 operations and reliability tasks in a simulated online store. The verifier compares system state before and after their work. Expert Aug 2026 Not in cards
dfbench Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately. Expert Aug 2026 69% Not in cards
DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. Everyday Aug 2026 71.4% Not in cards
Dirty Integration Agents collect every record from a paginated API despite unreliable cursors, rate limits, changing field types and timeouts. University Aug 2026 Not in cards
DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring. Expert Aug 2026 71.1% In use
DuMateBench Agents complete real-world productivity and office-work tasks in sandboxed workspaces with tools. Task evaluators check the requested artifact and record a reward. Expert Aug 2026 Not in cards
E-Bench-Code (Tencent) Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published. Expert Aug 2026 83.3% In use
E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. University Aug 2026 14.3 Not in cards
EarthVerse Agents investigate documented natural hazards using event data from different sources. The evaluator checks individual claims and the calculations that support them across 405 tasks. Expert Aug 2026 85% Not in cards
EduClaw-Bench Agents tutor a simulated learner over 30 days. The benchmark tracks learning gains, responsiveness, helpfulness, and curriculum design across 55 scenarios. University Aug 2026 Not in cards
EmbodiedCLUE-VLA RoboTwin Evaluation Robot policies perform ten two-handed manipulation tasks under randomized RoboTwin 2.0 Hard conditions. The score averages task success rates over twenty episodes per task. Everyday Aug 2026 Not in cards
Enterprise Task Pareto Frontier (Glean) Models complete 1,000 workplace tasks. A model compares paired answers for correctness, task completion, and readiness for use. Glean estimates preference ratings and cost per task. University Aug 2026 Not in cards
FactoryBench-100 Agents complete 100 manufacturing workflows through simulated ERP, email, spreadsheet, and chat tools. FactoryScore checks their decisions and the state left in each system. Expert Aug 2026 76.2% Not in cards
FinDeepIndicator Agents construct financial indicators by specifying formulas, retrieving market data, and calculating results. The evaluation checks performance at each step. University Aug 2026 39% Not in cards
Frontier Agentic ML & DS Agents solve machine learning and data science tasks in a terminal. Verifiers check their deliverables and predictive quality. Expert Aug 2026 64.6% Not in cards
Frontier Agentic STEM Agents produce scientific results in a terminal. Task verifiers recompute the expected results from the underlying data. Expert Aug 2026 55.1% Not in cards
FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. Expert Aug 2026 20.6% Not in cards
FrontierPhysics Agents do end-to-end physics research on 56 tasks that 42 physicists from 26 institutions built from their own work, across nine domains of theoretical, computational and experimental physics. Each task asks the agent to review the literature, plan, run the research and write a manuscript. Deterministic verifiers and expert rubrics grade whether the research is correct and publishable, over 3 trials. Expert Aug 2026 27.4% Not in cards
Game Arena Nine Men's Morris Language models play Nine Men's Morris. Forming a row of three, called a mill, lets a player remove one of the opponent's pieces. Everyday Aug 2026 1,190 Not in cards
Game Arena Reversi Language models play Reversi, also called Othello, flipping the opponent's disks that they flank in a line. Everyday Aug 2026 1,306 Not in cards
Game Arena Word Art Language models take turns drawing a word as ASCII art and guessing what the other model drew. Everyday Aug 2026 2,528 Not in cards
GameXpert-Bench Agents build games from 97 briefs, repair bugs in 100 tasks, and improve 17 games over six requests each. Each track reports its own scores. University Aug 2026 70.9 Not in cards
GDPevo Agents learn from training tasks drawn from enterprise workflows and must reuse that experience on related test tasks. Gains at test time measure self-evolution. Expert Aug 2026 67.1% Not in cards
Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. Expert Aug 2026 4.00% System cards only
Gray Swan Cyber Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on cyber tasks built from Google scenarios. Reports query and objective success rates. Expert Aug 2026 10% System cards only
Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. Expert Aug 2026 10.1% Not in cards
HarnessOpt-Bench Coding agents edit a target agent’s harness under a fixed evaluation budget. The grader measures improvement over the starting harness on held-out tasks. Expert Aug 2026 0.47 Not in cards
HEPToolBench 1.2 Models generate inputs for particle physics simulations and inspect the resulting files. The main suite has 28 tasks and a separate extension adds three debugging tasks. Expert Aug 2026 71.4% Not in cards
HM3D-MFMON Agents navigate across floors and find a sequence of objects. Everyday Aug 2026 Not in cards
HWE-bench CAE Agents run engineering simulations, extract numerical results, and report them. A verifier repeats the extraction from the solver output and checks engineering tolerances. Expert Aug 2026 0.90% Not in cards
Hy-BrowseComp-Pro2 Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published. Expert Aug 2026 56.1% In use
Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published. Expert Aug 2026 62.4% In use
Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. Expert Aug 2026 79.7% In use
Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. Expert Aug 2026 57% In use
Hy-LifeSearch Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published. Expert Aug 2026 49.2% In use
HybridDeepResearch Agents combine database queries and web search to answer research questions under constraints. Expert Aug 2026 32.8% Not in cards
HyDRA-Bench Agents interleave database queries and document retrieval to answer questions. A model judge checks the final answer. University Aug 2026 53.9% Not in cards
InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. Expert Aug 2026 46.9% System cards only
Insight Consolidation v1 Agents read customer account documents, infer why the customer acted, and cite supporting records. The evaluator combines a model judgment with citation accuracy. University Aug 2026 Not in cards
IronClad Agents repair security misconfigurations in Terraform repositories. The grader verifies that the repaired configuration satisfies policy and preserves the intended infrastructure. University Aug 2026 Not in cards
ITSMBench Agents operate an IT service desk across 53 conversations with simulated colleagues. The verifier grades the resulting database state, including permissions and required actions. University Aug 2026 0.70 Not in cards
KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. Expert Aug 2026 37.2% System cards only
LAB: Law Firm Knowledge Agents search a simulated law firm’s matter files and deliver work that satisfies legal task criteria. Expert Aug 2026 Not in cards
LedgerBench-100 Agents perform 100 corporate finance tasks through 66 tools in a shared enterprise system. SQL checks grade the changes they make. Expert Aug 2026 Not in cards
LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. Everyday Aug 2026 66.9% Not in cards
LLM Fantasy Football Arena Classic Models manage two NFL fantasy teams each. One league allows trades and messages. The other disables both. Each league scores its own draft and season. Everyday Aug 2026 0.90 Not in cards
LoopArena Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit. Expert Aug 2026 36.3% Not in cards
MineAmongUs Vision-language agents play an embodied Among Us game with verbal and nonverbal deception. Everyday Aug 2026 81.3% Not in cards
MIVAS Bench Voice agents handle calls for a dermatology practice, a law firm, and a retailer. Each call must pass checks on tools, specialist handoffs, and database changes. University Aug 2026 56.5% Not in cards
MobilePA-Bench Agents complete 1,705 mobile planning tasks using tools, persistent memory, reusable skills, and specialist agents. Task-specific checks evaluate tool calls, state changes, and interaction behavior. Additional gates check required memory access and skill loading. Everyday Aug 2026 76.9% Not in cards
MOMENT Eval Models use a calculator and Python to solve 91 structural engineering problems. They return a value or classification that a program checks against the answer key. Expert Aug 2026 95.6% Not in cards
NanoGPT Speedrun Frontier Agents change a GPT optimizer on fixed GPUs. Each candidate must reach the target validation loss across eight fixed seeds using fewer training steps. Research frontier Aug 2026 2726.0 Not in cards
NCP-Bench Models narrate interactive stories while preserving facts, commitments, and plot order against player interventions. A model auditor checks 100 movie-derived environments after each turn. The evaluation reports survival, conflicts, and narrative progress. Everyday Aug 2026 32.9 Not in cards
OfficeQA Pro V2 Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match. University Aug 2026 54.4% In use
OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. University Aug 2026 100% Not in cards
Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. University Aug 2026 60.4% System cards only
OmniMech Models write CAD programs to reconstruct parts from dimensioned drawings. The evaluator compares the reconstructed geometry with reference parts and measures relative volume error. University Aug 2026 15.9% Not in cards
OpenStudio-MCP LLM Test Benchmark v1.2.1 Agents change building energy models with OpenStudio tools. The grader checks the tool calls and inspects the resulting model against task requirements. University Aug 2026 100% Not in cards
OSWorkerBench Desktop agents complete long-horizon office tasks across business applications with executable evaluators and optional demonstrations. Expert Aug 2026 Not in cards
OSWorld 2.0 (Aug 2026) Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. University Aug 2026 77.9% System cards only
Palmyra X6 Enterprise Workflow Evaluations Agents perform production workflows that require retrieval, tool use, content generation, or delegation. Writer compares capability and cost. Expert Aug 2026 Not in cards
PlayWorld An agent player pursues objectives inside video world models. Gemini 3.1 Pro applies weighted rubrics for geometry, interaction, and state evolution across the resulting videos. Everyday Aug 2026 Not in cards
PosterBench Agents turn research papers into posters. Programmatic checks and visual model judgments score seven content and design dimensions, with ceilings for serious rendering or layout failures. University Aug 2026 Not in cards
PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. Expert Aug 2026 63.3% Not in cards
QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. University Aug 2026 69.9% In use
Retrobench v0.4 Agents reconstruct C from five Nintendo 64 assembly targets. The organizer scores the average percentage of matching bytes under a budget of 25 compilations per target. Expert Aug 2026 99.9% Not in cards
RevBench (Polyvor Labs) Agents analyze supplied artifacts in 22 local reverse engineering puzzles. The verifier checks each submitted answer for an exact match. Expert Aug 2026 50% Not in cards
RSI Bench Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks. Research frontier Aug 2026 38.9 Not in cards
RSI-Exam Agents get a weak method on a long research task and must turn it into one that scores better on hidden data. It measures recursive self-improvement on executable research. Research frontier Aug 2026 53.6% Not in cards
S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. Expert Aug 2026 Not in cards
SalesBench-100 Agents reconcile sales evidence, update authorized records, and deliver executive briefs across 100 workflows. Checks grade procedure, system state, and outputs. Expert Aug 2026 Not in cards
SecIT Bench Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions. Expert Aug 2026 82.9% Not in cards
SemaPLC Project-Context Track Agents implement plant control logic inside existing Structured Text projects. The evaluator measures compilation, static checks, and agreement with reference outputs during execution. Expert Aug 2026 100% Not in cards
Small Model Agent Bench Models choose tools and supply their arguments in 28 Chinese and English episodes. Fixed tool responses let the grader check planning, recovery, and task completion. School Aug 2026 95.7% Not in cards
Speech Agent Arena People hold live conversations with two hidden speech-to-speech agents on the same scenario, some with tool calls, then pick the one they preferred. Votes are fit to an Elo rating, and the arena also tracks task success. Everyday Aug 2026 1,070 In use
SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. Expert Aug 2026 56.9% In use
Sregym env Agents diagnose incidents in a running service with generated logs and traffic. The grader checks service recovery, the underlying fault and collateral damage. University Aug 2026 Not in cards
SWE Atlas Codebase QnA Extended Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores. Expert Aug 2026 44.7% Not in cards
SWE Refactor Bench Agents migrate 20 whole open-source repositories to a new language, framework, platform, or build system. Graded by migration audit, behavioral tests, and adversarial verifier models. Research frontier Aug 2026 47% In use
SWE-Bench ProMax Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions. Expert Aug 2026 41.2% Not in cards
SWE-bench Science Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix. Expert Aug 2026 50.4% Not in cards
SWE-bench Verified Extended Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores. Expert Aug 2026 83.3% Not in cards
SWE-Marathon-Ext Agents build 12 software products from specifications. Automated tests and browser checks grade the applications. Expert Aug 2026 74% Not in cards
Terminal-Bench 2.1 Extended Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores. Expert Aug 2026 54% Not in cards
Terminal-Bench 4.0 Agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. Expert Aug 2026 64.8% In use
Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. Expert Aug 2026 68.1% In use
Thinkingbox-bench Agents complete 507 business workflows through tools and simulated user conversations. Executable checks require correct backend updates without forbidden side effects. University Aug 2026 66.5% Not in cards
Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. Expert Aug 2026 71.6 Not in cards
TxBench-Antibody-Discovery Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision. Expert Aug 2026 67.7% Not in cards
Vals Index v2 Version 2 of Vals AI's GDP-weighted index, which brought back a legal sector and added financial modeling in Excel and code migration. Aug 2026 69.7% Not in cards
Vals RSI Index Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness. Research frontier Aug 2026 0.38 Not in cards
Vals Smith: abhishek203/SfAgent Agents modify abhishek203/SfAgent to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 80% Not in cards
Vals Smith: apache/spark Agents modify apache/spark to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 84.5% Not in cards
Vals Smith: arandomhooman/hoomans-morphe-patches Agents modify arandomhooman/hoomans-morphe-patches to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 20% Not in cards
Vals Smith: arduino/arduino-ide Agents modify arduino/arduino-ide to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 43.3% Not in cards
Vals Smith: astral-sh/uv Agents modify astral-sh/uv to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 66% Not in cards
Vals Smith: benredmond/apex Agents modify benredmond/apex to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 63.3% Not in cards
Vals Smith: BerriAI/litellm Agents modify BerriAI/litellm to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 70% Not in cards
Vals Smith: chrisanderton/tyk-gateway Agents modify chrisanderton/tyk-gateway to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 76.7% Not in cards
Vals Smith: conjfrnk/blackbird Agents modify conjfrnk/blackbird to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 56.7% Not in cards
Vals Smith: conjfrnk/memovox Agents modify conjfrnk/memovox to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 90% Not in cards
Vals Smith: flutter/flutter Agents modify flutter/flutter to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 50% Not in cards
Vals Smith: geopandas/geopandas Agents modify geopandas/geopandas to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 66.7% Not in cards
Vals Smith: goharbor/harbor Agents modify goharbor/harbor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 60% Not in cards
Vals Smith: jax-ml/jax Agents modify jax-ml/jax to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 79.6% Not in cards
Vals Smith: ladydascalie/core Agents modify ladydascalie/core to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 76.7% Not in cards
Vals Smith: leanprover-community/mathlib4 Agents modify leanprover-community/mathlib4 to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 76.7% Not in cards
Vals Smith: mirumee/ariadne Agents modify mirumee/ariadne to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 83.3% Not in cards
Vals Smith: nationalsecurityagency/ghidra Agents modify nationalsecurityagency/ghidra to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 73.3% Not in cards
Vals Smith: NVIDIA/nccl Agents modify NVIDIA/nccl to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 46.7% Not in cards
Vals Smith: oven-sh/bun Agents modify oven-sh/bun to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 54.2% Not in cards
Vals Smith: palantir/blueprint Agents modify palantir/blueprint to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 93.2% Not in cards
Vals Smith: poalrom/sharepaste Agents modify poalrom/sharepaste to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 50% Not in cards
Vals Smith: sashabaranov/go-openai Agents modify sashabaranov/go-openai to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 96.7% Not in cards
Vals Smith: sgl-project/sglang Agents modify sgl-project/sglang to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 78% Not in cards
Vals Smith: skypilot-org/skypilot Agents modify skypilot-org/skypilot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 30% Not in cards
Vals Smith: suitedaces/dorabot Agents modify suitedaces/dorabot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 66.7% Not in cards
Vals Smith: tock/tock Agents modify tock/tock to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 86.7% Not in cards
Vals Smith: torvalds/linux Agents modify torvalds/linux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 74.1% Not in cards
Vals Smith: vercel/next.js Agents modify vercel/next.js to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Expert Aug 2026 74.3% Not in cards
VideoGAIA Agents inspect selected video segments, search the web, and read pages before answering questions. Model judges check answers, and the harness reports accuracy by category and difficulty. Everyday Aug 2026 58.3% Not in cards
VWE-Bench Multimodal agents build interactive 3D worlds from a user request. They infer intent, plan the scene layout, call 3D tools for asset retrieval, editing and rendering, and revise from rendered feedback. The benchmark has 2,616 3D assets, 323 human-annotated seed worlds and 6,828 queries. Verified queries have ground truth, and the rest are scored with rubrics. Expert Aug 2026 57.3% Not in cards
WildArtifactBench Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo. Expert Aug 2026 Not in cards
WRENCH (Factorio) Agents build Factorio factories and recover from unannounced disruptions. The evaluator measures how much production the factory retains after a disruption. University Aug 2026 1.49 Not in cards
24/7 ClawBench 2.0 Agents do always-on assistant work through OpenClaw. Tasks span several days, events arrive at the same time, and interruptions are routine. Expert Jul 2026 48.3% System cards only
3D SubwayBench Agents use Command Code to build a Subway Surfers style endless runner in a single HTML file. The scorecard grades six features from the submitted code. University Jul 2026 5.00 Not in cards
AbstentionAct Agents handle HR and operations requests with missing information or authority. The benchmark checks whether agents clarify, abstain, or complete a permitted action. Expert Jul 2026 Not in cards
ActiveVision Models inspect images repeatedly to count regions, trace structures, and compare visual attributes. The released split has five examples for each of 17 tasks. Exact match checks the final answer after normalizing case, spacing, and separators. Everyday Jul 2026 50.6% Not in cards
Agent Behavior Bench Agents are scored on process quality, including tool-use behavior, efficiency and discipline, together with task completion. Expert Jul 2026 65% System cards only
Agent Memory Leaderboard Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework. Everyday Jul 2026 Not in cards
AgentHarm Verified (Meta) Agents receive 108 malicious AgentHarm tasks kept after human review, plus a benign set. Reports average harm score and false refusal rate. Expert Jul 2026 3.40% System cards only
Agentic Trader Agents search for market information and place simulated foreign exchange orders using live prices. Each agent manages its own positions, cash, and margin. University Jul 2026 79.4% Not in cards
Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. Expert Jul 2026 78.3% System cards only
AgenticDataBench Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios. Expert Jul 2026 48.8% Not in cards
AI Battle Arena Models play poker, Gomoku, and Connect Four through the same interface. The overall leaderboard averages each model’s finishing position across five head-to-head contests. Everyday Jul 2026 88.0 Not in cards
AISE-Bench Agents answer academic information-seeking questions through multi-step knowledge-graph APIs and source-grounded responses. Planning, execution and answer-grounding metrics score each trajectory. Expert Jul 2026 Not in cards
APEX-Accounting Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria. Expert Jul 2026 71.5% In use
ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. Everyday Jul 2026 47.6% Not in cards
Artificial Turf War Models bid for draft position, pick NFL players, and submit weekly lineups. The league ranks teams by wins and breaks ties with fantasy points. Everyday Jul 2026 6.00 Not in cards
ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. Everyday Jul 2026 System cards only
AutoCAD Bench Computer-use agents turn a dimensioned reference drawing into a correct, editable DWG in AutoCAD through the screen alone, with mouse and keyboard. 50 tasks span 2D drafting and 3D modeling at four difficulty levels. Expert Jul 2026 46% Not in cards
aws-bench Coding agents and model pairs diagnose misconfigurations, provision infrastructure and operate live cloud setups in disposable real AWS accounts. Expert Jul 2026 Not in cards
BackSearch Paid API that searches a frozen web archive as of a given date, for forecasting backtests and reproducible web-enabled evals. A tool, not a benchmark. Jul 2026 Not in cards
BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. Everyday Jul 2026 Not in cards
Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. Expert Jul 2026 72% In use
BioCircuit Agents identify groups of features in Qwen3-4B that control a behavior or restore a damaged output. The grader tests their interventions on held-out prompts. Research frontier Jul 2026 Not in cards
BioSecBench-Refusal Agents run 61 routine and 46 concealed-hazard biology analysis tasks. The score pairs red-team refusal with routine completion. Expert Jul 2026 62.4% System cards only
BioSecBench-Surveillance Agents turn raw pathogen sequencing data into surveillance results. The endpoint pass rate excludes refused runs. Expert Jul 2026 60.8% System cards only
Bot Analysis Arena Models write trading rules for simulated US stock accounts. Some agents revise their rules after trades. Software tests each revision against the current rules before adoption. University Jul 2026 Not in cards
Cekura Bench Models complete separately scored speech recognition, speech generation, voice-agent, real-call, turn-taking and voice-quality benchmarks. Cekura publishes no combined score. Everyday Jul 2026 Not in cards
Cekura Production Voice Agents Published composite score for production voice agents. Everyday Jul 2026 Not in cards
Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability. Expert Jul 2026 59.9% System cards only
Command Code Flappy Bird Agents use Command Code to build a Flappy Bird game with Canvas. The scorecard grades features including collisions, window resizing, and offline play. University Jul 2026 4.83 Not in cards
Command Code Pixel Art Space Shooter Agents use Command Code to build a scrolling Canvas game with cave walls and collectible gems. The scorecard grades features including collisions and offline play. University Jul 2026 4.83 Not in cards
Composite-Bench Agents solve generated browser tasks in planning, rule extraction, deduplication, routing, and information gathering. Success requires matching a certified optimal solution. A separate number-partitioning task measures solution quality rather than contributing to the success rate. Expert Jul 2026 74% Not in cards
Cost per Successful Task Agents solve 40 Terminal-Bench tasks. The benchmark divides spending across all attempts by the number of successful runs. Expert Jul 2026 $23 Not in cards
CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. Research frontier Jul 2026 34.2% Not in cards
CursorBench 3.2 Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.2 added instruction following and advanced tool use problems. Agentic graders score correctness. Expert Jul 2026 73.4% In use
CyberGym (final-submission metric) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. Expert Jul 2026 86.3% In use
Cybersecurity and Privacy Legal Agent Eval Agents write legal risk memos for five synthetic privacy and incident reporting matters. The grader checks the completed memo and its coverage of legal issues. Expert Jul 2026 Not in cards
Deep Research Bench (Moonshot) Agents answer deep-research queries curated by domain experts. Expert-aligned rubrics grade the reports. Expert Jul 2026 90% System cards only
DeepsecBench Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings. Expert Jul 2026 61.2 Not in cards
DexVerse Robot policies perform tabletop manipulation tasks in Isaac Lab, including tool use, bimanual coordination, and sequences with several goals. Task-specific success conditions check completion under different visual settings and robot embodiments. Everyday Jul 2026 34% Not in cards
DiligenceBench 150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness. Expert Jul 2026 58.1% In use
Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. Expert Jul 2026 97.3% Not in cards
E-Bench (Tencent) Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published. Expert Jul 2026 80.6% In use
EBR-bench Agent plays the board game Earthborne Rangers 10-30 times in a row. Scored on objectives completed (of 21) to test learning from experience. Everyday Jul 2026 92.4% Not in cards
EdgeBench Agents iterate for up to 12 hours on 134 executable real-world tasks with multi-level feedback. Scores are tracked at 2 hour steps to measure learning from the environment. Expert Jul 2026 51.3 Not in cards
Employment Law Legal Agent Eval Agents review employment case files and write risk memos. The grader checks whether the memos identify planted legal issues and avoid unsupported findings. Expert Jul 2026 Not in cards
Enterprise Vendor Eval Agents review vendor documents, question an internal IT team, and recommend whether to approve a purchase. The grader checks their decision and identified requirements. University Jul 2026 Not in cards
Enterprise-Bench L1-L2 Agents answer 14 operational questions across connected enterprise records. Tasks require either retrieving facts or combining evidence under business rules. University Jul 2026 82.9% Not in cards
EvoPolicyGym Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves. Expert Jul 2026 0.89 Not in cards
Excel Modeling Benchmark Agents build investment banking and private equity financial models in Excel from a written brief and source files. Rubrics give partial credit for tabs, structure, formulas and formatting. Expert Jul 2026 76.7% Not in cards
FACTS Search V2 Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle. University Jul 2026 89.9% Not in cards
FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. Jul 2026 84% System cards only
Feishu Office v1 Agents handle 140 Chinese workplace requests in a simulated Feishu workspace. The grader checks the resulting calendars, documents, tables, and approvals. University Jul 2026 Not in cards
Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. Expert Jul 2026 62.6% System cards only
Founder Bench Agents run four matched businesses per model through the acoco platform. The evaluation tracks real advertising, customer attention, and website traffic after seven days. Its headline customer-impressions metric adds ad impressions to organic site visits. Expert Jul 2026 127902.0 Not in cards
FrontierCode 1.1 Real-repo coding tasks graded for mergeability: correctness, regressions, lint, tests, scope and code quality. Failing any blocker scores zero. Expert Jul 2026 53.5% In use
FrontierFinance Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics. Expert Jul 2026 55.9% In use
Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. Everyday Jul 2026 Not in cards
Game Arena Bargaining Two language models negotiate how to split a pool of items, each with private item values. Each side tries to reach a good deal without giving away its values. Everyday Jul 2026 317 Not in cards
Game Arena Coin Game Two language models move on a grid and pick up coins. Each should collect the coins that score for it and avoid the rest. Everyday Jul 2026 514 Not in cards
Game Arena Lines of Action Language models play Lines of Action, where a piece moves as far as the number of pieces on its line. The goal is to join all of one's pieces into one group. Everyday Jul 2026 1,274 Not in cards
Game Arena Ultimate Tic-Tac-Toe Language models play nine linked tic-tac-toe boards. Each move decides which small board the opponent must play next. Everyday Jul 2026 720 Not in cards
GameEngineBench Agents edit C++ files in existing Unreal Engine 5 projects. Runtime tests check the changes, and a model audits whether the behavior meets the specification. University Jul 2026 55.5% Not in cards
GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. Everyday Jul 2026 Not in cards
Gauntlet Agents develop game controllers through practice. The authors freeze each controller and test it on unseen games without further model calls. University Jul 2026 Not in cards
Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. Everyday Jul 2026 Not in cards
Gorgias AI Agent Competitive Benchmark Agents recommend products and answer support questions on live Shopify stores. Gorgias combines answer quality, response time, and the share of conversations handled without human help. Everyday Jul 2026 Not in cards
HANDBOOK.md Agents do routine company work with mock email, chat, calendar, issue and commerce tools while following a 20 to 124 page handbook. Programmatic criteria check required and forbidden actions, and a trial passes only if all pass. University Jul 2026 52.3% In use
HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. Everyday Jul 2026 11.1% Not in cards
Hy-Backend 2.0 Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published. Expert Jul 2026 35.2% In use
Hy-CompanyBench Coding agents complete Tencent in-house company engineering tasks. Task and scoring details are not published. Expert Jul 2026 41.7% In use
Hy-Euler pro Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published. Expert Jul 2026 24.2% In use
Hy-FinModelBench Agents build financial models for Tencent in-house finance tasks. Task and scoring details are not published. Expert Jul 2026 69% In use
Hy-SkillsWorld Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published. Expert Jul 2026 45.8% In use
ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. Expert Jul 2026 System cards only
InfraBench Agents deploy, profile, debug, and optimize model workloads on AMD MI35X GPUs. Task-specific verifiers grade 20 engineering assignments. Expert Jul 2026 Not in cards
KAET (Kimi Autonomous Execution Tasks) Agents execute long-horizon tasks that simulate real user requests and enterprise system operations. Expert Jul 2026 83.5% System cards only
KAT Claw Bench Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables. University Jul 2026 85.5% In use
KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages. University Jul 2026 53.1% In use
KernelBench (kernelbench.com) Coding agents write GPU kernels for real model workloads, such as fused MoE layers, paged attention and FP8 matrix multiplies. The site groups tasks into suites: Mega (one decode task), CUDA (4 tasks) and Hard (6 tasks). A run counts only if it passes an audit of correctness. Scores are speedup over a baseline or percent of the hardware roofline. Expert Jul 2026 1.00 Not in cards
Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. Expert Jul 2026 38.9% System cards only
Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. Expert Jul 2026 System cards only
Ko-GDPval Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format. Expert Jul 2026 87.3% In use
LAB: Diligence Agent researches a synthetic M&A virtual data room (thousands of documents, up to ~80M tokens) and drafts a diligence memo scored against planted-issue rubric criteria. Expert Jul 2026 Not in cards
Lamina Product Bench Agents build four small applications under three instructions: code directly, plan first, or use Lamina skills. A model judge grades each submission. University Jul 2026 Not in cards
LaminaBench v1 Agents implement the same product brief with and without Lamina skills. The package README specifies ten briefs across five categories and 20 paired Harbor tasks. University Jul 2026 Not in cards
LMArcade Agents build browser games from the same prompt. Players try two games and vote before learning which models and coding tools made them. University Jul 2026 1,798 Not in cards
Long-Horizon-Terminal-Bench (LHTB) Terminal-Bench-style container tasks needing hundreds of steps. Subtasks graded deterministically for dense partial-credit reward. Solved at reward >=0.95. Expert Jul 2026 60.2% Not in cards
LoopsBench Agents complete 112 software tasks with dependencies among smaller development units. Executable tests measure resolved tasks and the fraction of units that pass. Expert Jul 2026 25% Not in cards
Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. Everyday Jul 2026 3.23 Not in cards
MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. Expert Jul 2026 59.5 Not in cards
Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. Expert Jul 2026 16.8% System cards only
MIRA Bench Agents carry out long multi-role enterprise tasks across several systems and decide when to delegate to subagents. Expert Jul 2026 64.1% System cards only
MM-ToolSandbox Visual tool-calling agents complete 258 multi-image, multi-turn scenarios across 16 application domains and more than 500 tools. Rubric and entity-diff checks score the runs. Expert Jul 2026 Not in cards
Multi-Agent Arena Risk Four commanders, people or anonymized models, trade sealed messages and then attack for territory on a 42-territory map. Models are ranked by their mean share of territory, where 25 is an even split. Everyday Jul 2026 38.8% Not in cards
Multi-Agent Arena Social Poker No-limit hold’em tournaments with table talk, where people and anonymized models share a table. Models are ranked by a rating fit on who finished ahead at each table. Everyday Jul 2026 1,553 Not in cards
NanoGPT Expenditure Horizon Agents optimize NanoGPT training. METR estimates the budget at which human contributors become more cost effective than the agents. Expert Jul 2026 Not in cards
Olam Labs Multi-Agent Arena People and anonymized models share a table in social strategy games: Diplomacy, Social Poker, Codenames and Risk. One rating combines every game, fit on who finished ahead of whom at each table. Everyday Jul 2026 1,544 Not in cards
OmniRouting Models route copper traces between pads on printed circuit boards. The evaluator counts required nets that connect without design rule violations. University Jul 2026 28% Not in cards
Online Experience (Moonshot) Agents produce the deliverable file types that real online users request most often, following the distribution of real usage. Everyday Jul 2026 77.9% System cards only
OptimBench Dispatch Agents route vehicles under capacity and time constraints, then revise their routes after a disruption. University Jul 2026 0.92 Not in cards
ORCA-bench Agents investigate simulated production incidents and identify their root causes. The benchmark contains 1,079 tasks, including 324 held-out tasks covering 31 incidents. Expert Jul 2026 29.6% Not in cards
ParliamentBench Models play Secret Hitler while deceiving, persuading, and reasoning under hidden roles. Everyday Jul 2026 81% Not in cards
PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. Expert Jul 2026 4.25 Not in cards
PCBWorld-Bench Agents route printed circuit boards in KiCad. A board earns a clean pass when all required connections are complete and KiCad reports zero design-rule errors. University Jul 2026 0.54 Not in cards
PMPP-Hard Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. Research frontier Jul 2026 0.71 In use
PostTrainBench 1.1 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Research frontier Jul 2026 41.8% In use
ProdBench (Tencent) Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3. Expert Jul 2026 23% In use
Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. Expert Jul 2026 99.8% System cards only
ReactBench Coding agents do 39 realistic React tasks in two tracks. Write React tasks come from merged pull requests and are checked by hidden behavior tests. Fix React tasks ask the agent to remove React performance, accessibility and quality issues. A task passes only if the tests pass and a pinned React Doctor scan finds no new graded issue. Expert Jul 2026 54.4% Not in cards
RH SWE-bench v1 Agents repair software across Red Hat ecosystem repositories and test each change against the issue requirements. Expert Jul 2026 63.3% Not in cards
RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. Everyday Jul 2026 27.6% In use
RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. Research frontier Jul 2026 Not in cards
RSIBench-Data Coding agents act as data-centric researchers. With a fixed Tinker post-training stack and budget, they refine training data from dev-benchmark feedback, and the final model is scored on held-out test benchmarks. Research frontier Jul 2026 32.6% Not in cards
SecRespond Agents investigate compromised-host snapshots and write incident analyses and remediation plans. A model evaluator applies expert checklists separately to detection and response planning. Expert Jul 2026 79% Not in cards
Silent Exit Models fix four typos without changing correct lines. A held-out test measures whether models learned to quit early to avoid penalties. Everyday Jul 2026 Not in cards
SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. Expert Jul 2026 61.1% System cards only
SpecDecode RL Agents choose how many tokens to draft during speculative decoding. An offline grader measures speedup over the target model while requiring identical output. Expert Jul 2026 Not in cards
Speko Speech-to-Speech Benchmark (10-turn call) Voice agents answer a weather request with a tool, handle an interruption, and recall an earlier detail. A language model grades task success. Everyday Jul 2026 1108% Not in cards
Speko Speech-to-Speech Benchmark (20-turn concierge call) Voice agents try to book a flight, handle a failed booking, and remember a changed date. Eight automated checks score their actions and replies. Everyday Jul 2026 1.00 Not in cards
Spider 2.0-AIFunc Agents generate SQL that uses AI functions to solve 465 tasks across 125 databases. The evaluator compares execution results with reference results. University Jul 2026 70.3% Not in cards
SREGym-Lite Agents diagnose and mitigate failures in running services on a Kubernetes cluster. Expert Jul 2026 81.3% Not in cards
SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials. Expert Jul 2026 53.8% In use
Task AW IT support Agents complete five support workflows in simulated IT systems. Task checks cover onboarding, offboarding, and cleanup of accounts and assets. University Jul 2026 Not in cards
Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search. Expert Jul 2026 81.6% Not in cards
Terminal-Bench 3.0 Agents complete 74 tasks across 7 domains in sandboxed terminal environments. Verifier scripts grade each trial as pass or fail. Expert Jul 2026 42.7% In use
Terminal-Bench Data Agents solve terminal tasks in machine learning, data analysis, and scientific computing. Task verifiers grade the outputs. Expert Jul 2026 48.6% Not in cards
Time Horizon Index: KSP Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions. Expert Jul 2026 91.3% Not in cards
trivial-prompt-bench Models answer a short request through a terminal agent. Quesma adds the API charge to the estimated cost of the developer's waiting time. Everyday Jul 2026 $0 Not in cards
UAV Operator Models manage simulated drone missions as weather and flight restrictions change. The simulator scores their decisions from logged aircraft and mission state. Expert Jul 2026 Not in cards
UniClawBench Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill. Everyday Jul 2026 47.5% In use
Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. Expert Jul 2026 57.5% Not in cards
VAmoS Bench Voice agents answer simulated bank customers and use tools to manage credit cards. A model checks each call against its transcript and recorded actions. University Jul 2026 62.3% Not in cards
VariantBench Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result. Expert Jul 2026 52.8% Not in cards
VerticalRTK Search and answer APIs answer questions about facts that change over time across seven real-world domains. An LLM judge compares answers with dated references. Expert Jul 2026 Not in cards
WANDR 500 wide-search tasks: find tens to hundreds of entities and facts, then return structured records with citations. Everyday Jul 2026 82.1% In use
WorkBuddyBench-Office Agents edit office files and reconcile records across a workspace. One task requires updating an eleven-slide presentation from a spreadsheet while preserving unrelated slides. University Jul 2026 86.4% Not in cards
WorkSurface-Bench Agents choose documents, tables, or graphs to answer workplace questions. The benchmark scores where agents look, what evidence they find, and their answers. University Jul 2026 10.3 Not in cards
ZendoWorld Models infer hidden rules in the board game Zendo from labeled examples. Everyday Jul 2026 Not in cards
ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 100% System cards only
ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 100% System cards only
ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 82.9% System cards only
Act2Answer Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control. Everyday Jun 2026 68% Not in cards
AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. Everyday Jun 2026 67.8% Not in cards
Agent Arena Arena ranks models as the orchestrator in real Agent Mode sessions on arena.ai. Scores use behavioral signals such as file downloads, disapproval events, retries and steerability. Everyday Jun 2026 28.6% In use
Agent Startup Bench Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews. Expert Jun 2026 68.8% In use
Agentic Abstention Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on. Everyday Jun 2026 Not in cards
AgenticSTS Agents play the Silent in Slay the Spire 2 at its lowest difficulty. The authors compare wins with different memory settings and two other agents. Everyday Jun 2026 60% Not in cards
Agents' Last Exam 1,500+ expert-sourced tasks from real projects across 55 occupations in three difficulty tiers. Objective verifiers grade deliverables. 5-hour cap. Expert Jun 2026 38.2% In use
Artificial Analysis Coding Agent Index v1.1 Version 1.1 of Artificial Analysis's coding agent index, which replaced SWE-Bench-Pro-Hard-AA with DeepSWE. Jun 2026 80.0 In use
Artificial Analysis Intelligence Index v4.1 Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents. Jun 2026 61.0 In use
ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. Everyday Jun 2026 48.2% Not in cards
AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. Expert Jun 2026 0.68 Not in cards
AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. Expert Jun 2026 66.5 Not in cards
BabaIsBench Agents play levels of the puzzle game Baba Is You through a text interface and win by rewriting the rules. Models are compared on pass rate, speed and cost. Expert Jun 2026 100% Not in cards
BigFinanceBench Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference. Expert Jun 2026 58.8% In use
BIM-Edit Agents create, change, or delete elements in IFC building models. The evaluator checks the resulting geometry, attributes, and relationships. University Jun 2026 87.7% Not in cards
BTF-3 Agents forecast already-resolved binary and numeric questions while researching a frozen snapshot of the web. Scoring pools Brier and ranked probability scores. Expert Jun 2026 0.12 Not in cards
BuildArena 2.0 Construction Challenge Agents build and test machines in Besiege for seasonal engineering challenges. The organizers score each entry using simulation results and the rules for its track. Competition Jun 2026 Not in cards
bun-server-bench Agents implement 50 backend services in Bun. Hidden tests grade behavior such as transaction handling, concurrent requests, and rate limits. Expert Jun 2026 Not in cards
CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. Expert Jun 2026 $22,148,357 Not in cards
ChainWorld Agents complete sequences of desktop tasks while preserving the state needed for later steps. OSWorld checks each step. Everyday Jun 2026 31% Not in cards
Claw-SWE-Bench General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches. Expert Jun 2026 79.8% Not in cards
Code Migration Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. Expert Jun 2026 69.8% Not in cards
CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. University Jun 2026 Not in cards
Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. Expert Jun 2026 0.20 Not in cards
CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. Everyday Jun 2026 42.5% In use
Cua-Bench EDA (KiCad v1) Agents build or edit KiCad schematics through the desktop interface. The grader checks the resulting netlist and reports full passes, partial passes, and mean reward. Expert Jun 2026 24% Not in cards
CyberBench (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Expert Jun 2026 88.1% Not in cards
CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. Expert Jun 2026 89% Not in cards
CyberGym-E2E (615 tasks) Agents discover and demonstrate vulnerabilities on the older 615-task cohort. The headline is cumulative stage S3 success. Expert Jun 2026 22.6% Not in cards
DailyReport Search agents answer practical daily information requests. Evaluators assess the resulting reports across dimensions of user satisfaction. Everyday Jun 2026 2.89 Not in cards
DeskCraft Desktop agents complete 538 Ubuntu tasks, including 152 tasks with later user clarifications or interruptions. Programmatic checks inspect final application state, saved files, and exported artifacts. Expert Jun 2026 29.8% Not in cards
DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. Expert Jun 2026 71.5% Not in cards
DuetBench Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head. Expert Jun 2026 1,035 Not in cards
DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. Everyday Jun 2026 10.6% Not in cards
EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. Everyday Jun 2026 Not in cards
Eleusis Agents play cards to discover a hidden rule. The scorer tests whether a proposed rule makes the same decisions as the hidden rule. Competition Jun 2026 82.2% Not in cards
EnterpriseClawBench Agents do 852 tasks rebuilt from real proprietary workplace agent sessions, each with fixtures, hard rules and semantic rubrics. Expert Jun 2026 66.3% Not in cards
EpiBench Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result. Expert Jun 2026 39.6% Not in cards
Evalatro v2 Models play Balatro from the game state and try to clear Ante 12. Illegal moves reduce the score for progress through the game. Everyday Jun 2026 46.1 Not in cards
EvalConvoLearn Models simulate learners in tutoring conversations. The evaluator compares their answers, mistakes, questions, and turn lengths with patterns in real student conversations. University Jun 2026 Not in cards
EvoArena Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes. Expert Jun 2026 53.3% Not in cards
EvoArena SWE-Chain-Evo Agents solve software tasks that change across versions. The score is the published accuracy across versioned tasks. Expert Jun 2026 Not in cards
EvoArena Terminal-Bench-Evo Agents solve terminal tasks that change across versions. The score is the published accuracy across versioned tasks. Expert Jun 2026 65.1% Not in cards
EvoCode-Bench v2 Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit. Expert Jun 2026 59.1% Not in cards
FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. University Jun 2026 77.8% In use
FrontierCode Real-repo coding tasks graded for mergeability: correctness, regressions, lint, tests, scope and code quality. Failing any blocker scores zero. Expert Jun 2026 34.3% In use
FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. Research frontier Jun 2026 38.1% In use
Game Arena Checkers Language models play American checkers against each other, with forced jumps and crowned kings. Ratings come from head-to-head games. Everyday Jun 2026 1,382 Not in cards
Game Arena Clobber Language models play Clobber, where every move captures an adjacent enemy stone. The player who makes the last legal move wins. Everyday Jun 2026 819 Not in cards
Game Arena Dark Hex Language models play Hex without seeing the opponent's stones, so they must probe cells to learn the hidden board while building a path. Everyday Jun 2026 581 Not in cards
Game Arena Dots and Boxes Language models take turns drawing lines between dots on a grid. Closing the fourth side of a box claims it. Everyday Jun 2026 1,393 Not in cards
GameCraft-Bench Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together. Expert Jun 2026 75.9% Not in cards
GDPval-AA v2 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Outputs are compared pairwise and ranked by Elo. Expert Jun 2026 1,861 In use
GeneBench Pro Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. Expert Jun 2026 31.5% In use
Harbor-Index Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. Research frontier Jun 2026 28.1% In use
HealthAgentBench Agents use terminal tools to complete 54 clinical and biomedical tasks across seven categories. Task-specific verifiers measure success, including a model judge for radiology report correction. Expert Jun 2026 55% Not in cards
Hedge-Bench Agents analyze supplied company and investment documents for 102 open-ended financial research tasks. A model judge applies rubrics that check whether the reasoning covers the analytical steps identified by investment professionals. Expert Jun 2026 31.5 Not in cards
IMO-LeanProofBench Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof. Competition Jun 2026 11.6% Not in cards
iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. Everyday Jun 2026 51.9% Not in cards
K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. Expert Jun 2026 45.7% Not in cards
KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. Research frontier Jun 2026 76.9% System cards only
Kimi Claw 24/7 Bench Agents work on 17 persistent multi-day professional scenarios through OpenClaw, with 610 checkpoints. The score is the mean pass rate over three runs. Expert Jun 2026 46.9% In use
Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages. Expert Jun 2026 77.1% In use
Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. Expert Jun 2026 56.3% In use
LLM SoccerArena World Cup 2026 Models forecast World Cup match scores and outcome probabilities before kickoff. The organizers compare forecasts with match results under different prompts and web access rules. Everyday Jun 2026 168.0 Not in cards
LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. Expert Jun 2026 0.66 Not in cards
MANDOS Run 003 Models make decisions during seven industrial incidents and four fictional scenarios. A model judge checks their decisions, including whether they seek expert advice. Expert Jun 2026 100% Not in cards
MarineEVT Models answer 2,000 held-out questions about underwater events, including species interactions, movement, and timing. The authors report accuracy across five reasoning categories. University Jun 2026 43.7% Not in cards
MCPMark Verified Agents complete 127 tasks through version-pinned MCP servers. Stabilized verifiers score single-run task success. University Jun 2026 96.1% Not in cards
MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. Expert Jun 2026 69.2 Not in cards
MetroLLM-Bench Agents operate a transit kiosk using tools for routes, fares, and service disruptions. The first score checks their actions automatically. The composite also includes model judgments. Everyday Jun 2026 92.0 Not in cards
MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. Research frontier Jun 2026 97.8% Not in cards
MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. Everyday Jun 2026 Not in cards
MyPCBench Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks. Everyday Jun 2026 62% Not in cards
NatureBench Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results. Research frontier Jun 2026 23.3% Not in cards
OmniGameArena Models play games while response latency advances the game clock. Everyday Jun 2026 0.47 Not in cards
Opus Magnum Bench Agents build machines on a hexagonal grid to make a target molecule. Each puzzle score compares the machine’s combined cost, cycles, and area with the human record. Competition Jun 2026 25.5 Not in cards
OSWorld 2.0 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. University Jun 2026 20.6% In use
PhoneHarness Bench Agents complete Android workflows through graphical controls, device commands, and structured tools. Checks inspect files, system settings, application state, and unintended side effects. Everyday Jun 2026 62.1% Not in cards
PlanBench-XL Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures. Expert Jun 2026 77.1% Not in cards
Poker Arena Models play no-limit Texas Hold’em across repeated sessions. The benchmark scores tournament chips and nine strategic dimensions, including memory and bluffing. Everyday Jun 2026 15730.0 Not in cards
Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. Expert Jun 2026 94.6% System cards only
PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. Expert Jun 2026 95.3% Not in cards
Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. Expert Jun 2026 76.9% Not in cards
Questflow Traders Arena Agents and people trade on Hyperliquid and Polymarket in a public competition. The leaderboard reports weekly portfolio returns alongside gains and trading volume. University Jun 2026 Not in cards
Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. Expert Jun 2026 62.1% Not in cards
Ramp SWE-Bench v1 Agents modify Ramp’s backend code to satisfy engineering requests. Hidden tests check whether each patch solves the task. Expert Jun 2026 88.5% Not in cards
ReactSim-Bench Driving simulators predict how traffic responds when the vehicle changes its behavior. The authors test those reactions in 2,636 scenarios. University Jun 2026 Not in cards
RedlineBench v2 (Fa1zvn) Agents negotiate eight contract terms with a simulated vendor. The grader compares the agreement with the best deal both sides could accept. University Jun 2026 Not in cards
Reward Hack Bench Agents attempt eight coding and puzzle tasks under controlled conditions. A judge distinguishes independent solutions from use of published answers. The control omits hints. University Jun 2026 Not in cards
Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. Expert Jun 2026 3.90% System cards only
RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. Everyday Jun 2026 49.9% Not in cards
Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. Everyday Jun 2026 Not in cards
RTSGameBench Agents play the open real-time strategy game Beyond All Reason. Expert Jun 2026 Not in cards
scBench-Long Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning. Expert Jun 2026 36.9% Not in cards
scBench-Long (original study) Agents complete 21 tasks in long single-cell data analysis workflows. The authors compare their answers with results from the original analyses. Expert Jun 2026 25.4% Not in cards
SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. Expert Jun 2026 58.4% Not in cards
SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. Everyday Jun 2026 66.6% In use
Senior Engineer Benchmark Agents rewrite the collaboration system in a frozen copy of the Proof codebase. Every grades their changes against a 100-point rubric. Expert Jun 2026 91.0 Not in cards
Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate. Expert Jun 2026 39.3% Not in cards
SentinelBench Agents monitor changing web environments and respond when scripted events make progress possible. Scores measure completion, reaction time and resource use. Expert Jun 2026 Not in cards
Slay the Spire Bench (alavico) Models choose an action from the text state of Slay the Spire. The campaign report measures floors reached and wins at each difficulty. Everyday Jun 2026 51.0 Not in cards
Snowflake Analytical Search: ArXiv Papers Agents count, group and compare findings across research papers. The evaluation checks numeric and categorical answers against references and scores lists with F1. University Jun 2026 Not in cards
SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. Everyday Jun 2026 Not in cards
SoCRATES Models mediate disputes in 600 scenarios across eight domains. A model judge rates topic agreement to measure consensus gain, intervention timeliness, and intervention effectiveness. University Jun 2026 Not in cards
SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. Everyday Jun 2026 14.4% Not in cards
SpatialWorld Digital Agents complete entertainment tasks in digital environments. The score is the published digital task success rate. Everyday Jun 2026 39% Not in cards
SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. Everyday Jun 2026 14.4% Not in cards
SpreadsheetBench 2 Agents complete 321 expert-annotated business spreadsheet workflows, such as financial modeling, debugging, and charting, in multi-sheet workbooks. Scoring is task accuracy. Expert Jun 2026 34.9% In use
STATE-Bench Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. Expert Jun 2026 58.9% Not in cards
Surface Evolver Bench Agents write Surface Evolver datafiles that model liquid surfaces under physical constraints across 16 tasks. Deterministic checks give partial credit for matching hidden reference quantities. Expert Jun 2026 95% Not in cards
SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail. Expert Jun 2026 21.5% Not in cards
SWE-Interact Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session. Expert Jun 2026 50.7% Not in cards
SWE-Marathon 20 realistic multi-hour software tasks (library reproductions, product clones, ML work). Verifiers combine hidden tests, parity and integrity checks. Expert Jun 2026 29% In use
SWE-Together Coding agents work through 109 repository tasks rebuilt from real multi-turn sessions between users and coding agents. A simulated user replays the original user's goals and gives feedback when the agent needs it. Scores count tasks solved and the number of corrections the user had to make. Expert Jun 2026 69.3% Not in cards
Terminal-Bench Challenges Agents build a large project end to end with no time limit, such as a faster Rust compiler or a WebGL renderer in WebAssembly. Each challenge has its own test bed and metric. Research frontier Jun 2026 Not in cards
Toolathlon-Verified 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. University Jun 2026 78.4% In use
ToolMaze Agents complete tool-use tasks while the runtime introduces explicit or hidden tool failures. A judge checks completion across four task structures and five perturbation settings. Metrics report task success, recovery rate, and recovery cost. University Jun 2026 53% Not in cards
Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. Expert Jun 2026 System cards only
Travel Agent Compassion Agents book travel activities when the customer does not mention animal welfare. The benchmark measures how often they choose a welfare-neutral option. Everyday Jun 2026 100% Not in cards
TUA-Bench Agents use a terminal for general computer tasks beyond coding, such as file, system and everyday productivity work. Everyday Jun 2026 65.8% Not in cards
Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. University Jun 2026 60.2% Not in cards
TxBench-Preclinical-Pharmacology Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision. Expert Jun 2026 68% Not in cards
UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. Everyday Jun 2026 55.8% Not in cards
VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output. Expert Jun 2026 50.1% In use
Vision2Web S3-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Expert Jun 2026 74.6% Not in cards
WeaveBench Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts. Expert Jun 2026 41.2% Not in cards
WebExploitBench Agents find and exploit 110 vulnerabilities in 15 real web applications, each in its own container. A verifier checks that the agent's proof of concept hits the same endpoint as the reference exploit. Expert Jun 2026 36.4% In use
τ-Rec Agents interview simulated moviegoers and search a catalog. A recommendation succeeds when it meets the user’s constraints and follows the service policy. Everyday Jun 2026 57.1% Not in cards
3DCodeBench Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. May 2026 55.4% In use
AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. Expert May 2026 0.80 System cards only
Acme Agent Evals Agents use GitHub tools to answer questions about issues, pull requests, and code in a simulated software project. University May 2026 85.1% Not in cards
AEC-Bench (Theodore Galanos) Agents perform engineering calculations such as cable sizing and hydraulic design. Task verifiers score their outputs, and the leaderboard reports mean reward and completion. University May 2026 0.89 Not in cards
AerialSim Agents fly a simulated drone using camera observations or write a program for the flight. The simulator checks the recorded trajectory against the mission requirements. Expert May 2026 73.2% Not in cards
Agent Island Models play multi-agent games that require cooperation, conflict, and persuasion. Everyday May 2026 0.88 Not in cards
AgentEscapeBench Agents solve procedural escape rooms using unfamiliar tools. The evaluator checks whether they complete dependent steps and escape. Everyday May 2026 76.9% Not in cards
Agentick Agents solve procedurally generated gridworld tasks involving navigation, planning, reasoning, memory, generalization, and coordination. Scores normalize task performance between random and oracle reference policies. Everyday May 2026 0.63 Not in cards
AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. Expert May 2026 38.4% In use
AgentKernelArena Coding agents optimize GPU kernels through full workflows with compilers and profilers. Scores check speedups and whether gains generalize beyond tuned shapes. Expert May 2026 3.23× Not in cards
AgentThreatBench Agents carry out routine tasks in the presence of untrusted content. The benchmark scores task completion and security separately. University May 2026 Not in cards
AI-Spire (AGI-Eval) Models replay one Slay the Spire 2 seed, build decks, and choose combat moves. The organizers record the floor reached on each attempt. Everyday May 2026 21.6 Not in cards
Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. May 2026 1.13 System cards only
Artificial Analysis Coding Agent Index v1.0 Artificial Analysis's first index of coding agents, each a model in a harness, averaged over three equally weighted coding benchmarks. May 2026 Not in cards
ASPI Agents complete simulated tasks with incomplete instructions. The authors compare adherence to user intent across matched versions of each task. Everyday May 2026 17.8% Not in cards
AsyncTool Agents coordinate several tool-use tasks while tool responses arrive after simulated delays. The evaluator checks function calls, parameters, task completion, and the final environment state. Scheduling metrics measure whether agents use waiting time effectively. Everyday May 2026 38.1% Not in cards
Augment Prism multi-turn coding benchmark Coding agents follow a queued conversation and edit a repository. A model compares the final changes with the original pull request. University May 2026 0.21 Not in cards
AvalancheBench (May study) Agents recover underlying patterns from synthetic enterprise data and report their findings. Expert May 2026 Not in cards
BiomniBench Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer. Expert May 2026 63.9 Not in cards
Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. Expert May 2026 System cards only
BoardGameBench (Homer Quan) Models play seven board game variants against a search engine. The scorer rewards quick wins and gives partial credit for longer losses. Everyday May 2026 264.5 Not in cards
CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. University May 2026 45.1% System cards only
Cattle Trade Agents play a long economic game with auctions, hidden offers, bargaining, bluffing, and resource limits. Everyday May 2026 Not in cards
CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. Expert May 2026 54.7% Not in cards
ClawBench V2 Browser agents do 130 new everyday tasks on live websites under six supported harnesses. Request interception and a more lenient LLM judge score each run. Everyday May 2026 44.6% Not in cards
Cohere North internal evals Cohere internal evals of its North agent platform: agentic question answering, spreadsheet data analysis and memory use. LLM-as-judge scoring. University May 2026 Not in cards
ContractBench (Observation Contracts) Agents complete 33 API workflow tasks. Validators check whether agents preserve returned values exactly and use them while they remain valid. University May 2026 77.8% Not in cards
CTF-Archive Diamond Agents solve 285 difficult capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. Expert May 2026 71% Not in cards
CursorBench 3.1 Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.1 added codebase understanding, bug finding, planning and code review problems. Agentic graders score correctness. Expert May 2026 72.9% Fading
CutVerse Agents complete editing tasks in seven applications, including Premiere Pro, Photoshop, and DaVinci Resolve. Vision models grade progress against milestones in the recorded workflow. University May 2026 68.3% Not in cards
DataPrep-Bench Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score. Expert May 2026 48.4 Not in cards
DataPrep-Bench Finance Models prepare training data for the finance domain. The score is the published downstream average after backbone training. Expert May 2026 59.3 Not in cards
DataPrep-Bench General Models prepare training data for the general domain. The score is the published downstream average after backbone training. Expert May 2026 78.2 Not in cards
DataPrep-Bench Law Models prepare training data for the law domain. The score is the published downstream average after backbone training. Expert May 2026 76.4 Not in cards
DataPrep-Bench Math Models prepare training data for the math domain. The score is the published downstream average after backbone training. Expert May 2026 25.5 Not in cards
DataPrep-Bench Medical Models prepare training data for the medical domain. The score is the published downstream average after backbone training. Expert May 2026 40.5 Not in cards
DataPrep-Bench Science Models prepare training data for the science domain. The score is the published downstream average after backbone training. Expert May 2026 26.8 Not in cards
DeepSWE 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. Expert May 2026 74.1% In use
DeepTerminalBench Agents get one attempt at each of 50 deep engineering tasks in Docker, in the Terminal-Bench 2.0 format. A task passes only if every hidden test passes. Tasks were calibrated so Claude Opus 4.6 still fails some at pass@4. Expert May 2026 34% Not in cards
DTap Direct Misuse An attacker agent generates adaptive jailbreaks against a target agent across tool-use and browser-use tasks. Reports attack success rate. Expert May 2026 21.2% System cards only
DTap Prompt Injection An attacker agent plants indirect prompt injections in tool and browser environments against a benign-user agent. Reports attack success rate. Expert May 2026 8.60% System cards only
ERP-Bench Agents complete 300 procurement and manufacturing workflows in Odoo 19. A constraint solver generates the tasks and optimal solutions, and the verifier checks final business state. Expert May 2026 43.4% Not in cards
ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. Everyday May 2026 70% Not in cards
euchre-bench Coding agents build a euchre bot and test it against four opponents. Each opponent contributes up to 100 wins to the total. University May 2026 140.0 Not in cards
EVA-Bench Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes. Everyday May 2026 0.74 In use
EvoCode-Bench Coding agents keep one project working across 26 tasks of 5 to 15 rounds each, while requirements change, pile up and conflict. Cumulative hidden tests check every round in a persistent container. Expert May 2026 54% Not in cards
ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. Expert May 2026 100% In use
ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. Expert May 2026 42.4% In use
Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. Expert May 2026 57% System cards only
Game Arena Word Association Teams of language models give one-word clues and guess linked words, in a game like Codenames, against an opposing team. Everyday May 2026 734 Not in cards
GBAEval Agents get 24 hours to write a Game Boy Advance emulator in Rust for WebAssembly. Replay, hardware, and audio tests compare it with Mesen2. Research frontier May 2026 90.6% Not in cards
Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. May 2026 0% System cards only
harness-bench-fast Agents operate on files and code using tools and memory. Harbor exports 231 tasks from version 0.3.0. Later public leaderboards report Light and Medium suites separately. University May 2026 Not in cards
HarnessAudit-Bench Agents run inside execution harnesses such as Claude Code, Codex and OpenClaw on 210 tasks across eight domains, from finance to software engineering, with safety rules built in. The audit reads the full trajectory through channels the agent cannot change. It scores whether the agent stays inside its permission and information-flow policies, completes checkpoints, and keeps both under prompt injection, vague goals and tool errors. Expert May 2026 0.41 Not in cards
Harvey LAB (Legal Agent Benchmark) Agent works through legal matter files across 24+ practice areas and produces work product. Expert rubric, task passes only if every criterion passes. Expert May 2026 25.4% In use
HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. Everyday May 2026 Not in cards
HWE Bench V1 (RISC-V CPU design) Agents improve a RISC-V processor in Verilog. Designs must pass correctness checks before the benchmark ranks them by estimated CoreMark iterations per second. Expert May 2026 983.2 Not in cards
HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials. Expert May 2026 67.6% System cards only
ITBench-AA Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON. Expert May 2026 Not in cards
JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. Expert May 2026 61.4% In use
Lazy investigation on misleading codebases (Anthropic) Models trace small, intentionally misleading codebases to answer how a value is set. Score is the share of lazy investigations ending in a wrong answer. Expert May 2026 0% System cards only
Local LLM Tetris Test Agents build Tetris in one HTML file and can revise it after playing. A separate Claude agent checks piece movement, line clearing, scoring, and restart. University May 2026 11.0 Not in cards
LongMemEval-V2 Memory systems help agents learn the affordances, dynamics and failure modes of customized web environments. Questions check whether that experience was internalized. Expert May 2026 1.30% Not in cards
Maze Bench Agents explore connected rooms in a 3D puzzle game. Each of the 100 gems contributes one percentage point to the score. Expert May 2026 24% Not in cards
MBABench Agents build complete financial models in Excel from a case prompt and documents. An LLM judge grades accuracy, formulas and formatting on a 0 to 100 scale. Expert May 2026 31.2 Not in cards
Megaminx solver Models solve a simulated twelve-faced twisty puzzle. They read the puzzle state as text, make moves with tools and receive rewards from the simulator. University May 2026 Not in cards
MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. Competition May 2026 63.3% System cards only
MLS-Bench Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Research frontier May 2026 50.3% In use
MobileGym Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state. Everyday May 2026 58.8% Not in cards
MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. Expert May 2026 16.2% Not in cards
NanoGPT-Bench Coding agents with a fixed H100-hour budget and no internet try to improve on a strong NanoGPT speedrun starting point. The score is the share of later human progress recovered. Research frontier May 2026 9.30% Not in cards
NARRA-Gym Agents build interactive stories around a user’s experiences and adapt scenes as the conversation unfolds. Judges rate story quality and user experience separately. Everyday May 2026 2 Not in cards
OpenComputer Computer-use agents complete 1,000 desktop tasks across 33 real applications, including browsers, office tools, creative software, development environments and messaging apps. App-specific verifiers inspect the final application state, and the harness gives partial credit from the recorded trajectory. Everyday May 2026 68.3% Not in cards
Parametric CAD Bench V1 Agents write FreeCAD scripts that build editable parametric parts from plain-English descriptions and dimension tables. Each of 100 tasks scores the harmonic mean of geometry similarity and spec consistency. Expert May 2026 In use
PDEAgent-Bench Models write finite element solvers for partial differential equations. A solver must run successfully, meet a numerical error tolerance, and finish within its time limit. Expert May 2026 54.1% Not in cards
PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. Expert May 2026 50% System cards only
PinchBench 2.0 Models drive an OpenClaw agent through 148 real-world tasks, including data and log analysis, meeting notes, DevOps, PDFs, research and writing. Graded by checks and a Haiku judge. Everyday May 2026 93.5% Not in cards
ProgramBench 200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original. Research frontier May 2026 18.5% In use
Prompt injection live bug bounty (Anthropic with Gray Swan) Expert red teamers submit prompt injection attacks against hidden frontier models in tool use, coding and computer use scenarios. Score is attack success rate. Expert May 2026 0.08% System cards only
PTCG-Bench Agents play the Pokémon Trading Card Game by choosing legal actions from a partially visible board. Match results determine their Glicko-2 ratings. School May 2026 1,854 Not in cards
QF-Bench Agents write and run code for 87 quantitative finance tasks in a network-isolated sandbox with real data. Tasks cover derivatives pricing, fixed income, credit, risk, factor research, backtesting and market microstructure. More than 3,300 assertions check numeric outputs against finance rules such as no-arbitrage bounds and put-call parity. Scored by pass@1 over three runs. Expert May 2026 61.7% Not in cards
ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. Research frontier May 2026 38.6 Not in cards
RoadmapBench Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests. Expert May 2026 39.1% In use
RoboChallenge Table30 v2 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. Everyday May 2026 0.73% Not in cards
RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. Everyday May 2026 21.5% Not in cards
Rule2DRC Models write KLayout scripts from chip design rules. The evaluator checks whether each script flags the correct violations on every test layout. Expert May 2026 67.9% Not in cards
SaaS-Bench 106 long-horizon GUI workflows across 23 locally deployed real SaaS apps. Scored from app databases with weighted checkpoints and full-resolution rate. University May 2026 31.1% In use
ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change. Expert May 2026 Not in cards
Science Sim Chemistry Models compare simulated compounds and choose whether to advance or reject them. The evaluator scores decisions against measured reaction outcomes and a budget. University May 2026 Not in cards
SEC-bench Pro Agents hunt and reproduce security bugs in large software projects from limited information. The first release covers V8 and SpiderMonkey. Expert May 2026 90.5% In use
SIGHT (Search and Intent Gauge for Hospitality and Tourism) Agents recommend restaurants, hotels and attractions from user preferences and search results. Evaluators check constraints, evidence and privacy. Everyday May 2026 Not in cards
Skill Reward Hacking Researchers track when models learn to increase a proxy reward without improving task quality. They measure warning time, false alarms, and transfer across reward functions. University May 2026 13.8 Not in cards
Slitherlink (savi) Models place edges to form a single loop on seven-by-seven Slitherlink boards. The evaluation uses ten boards disjoint from its 15 training examples. School May 2026 Not in cards
SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. Expert May 2026 40.2% Not in cards
SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. Expert May 2026 40.5% Not in cards
SpatialBench-Long (original study) Agents complete 24 tasks in long spatial transcriptomics workflows. The authors compare their answers with results from the original analyses. Expert May 2026 11.1% Not in cards
STATE-Bench Main 0.7.0 Agents complete enterprise tasks in travel, customer support, and shopping. The score is overall pass@1 for this track and version. Expert May 2026 55.7% Not in cards
Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses. Expert May 2026 67.1% Fading
SWE Atlas Refactoring Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result. Expert May 2026 59% In use
TeachArena Agents make teaching decisions, tutor students through dialogue, and carry out learning management system tasks. The evaluator checks conversation turns, artifacts, and final system states. University May 2026 0.80 Not in cards
Terminal-Bench 2.1 89 hard terminal tasks across 16 categories in containers. Hidden tests verify the final state. 5 trials per task. Expert May 2026 87.4% In use
Terminal-X A suite of three terminal datasets for coding agents. DeepTerminalBench tests single-shot depth, EvoCode-Bench tests multi-round iteration and RoadmapBench tests version upgrades on real repositories. Expert May 2026 42.5% Not in cards
TerminalWorld Agents complete terminal workflows reconstructed from public recordings in isolated environments. Automated tests check the resulting files and system state. A manually reviewed 200-task subset supports the published model comparisons. University May 2026 62.5% Not in cards
TERMS-Bench Agents negotiate the price of a single item with simulated counterparts over up to 10 rounds of alternating offers. Counterparts follow six behavior families across three regimes. A program scores each agent from its logged actions on four separate axes: surplus captured, agreement calibration, belief error about the opponent and procedural violations. Extra modes score merchant profit and a bankroll over many sessions. Everyday May 2026 0.69 Not in cards
TVRBench Agents move through simulated indoor scenes to reproduce a target image's viewpoint. Success requires stopping at exactly the target pose on the simulator's discrete grid. The evaluation split has 500 tasks across 80 scenes. Everyday May 2026 12% Not in cards
UniGUI-Bench Agents interpret and operate graphical interfaces. The benchmark tests 11 capabilities and includes separate system evaluations. Everyday May 2026 80.9% Not in cards
Vals Finance Agent v2 927 expert-reviewed research questions on public-company filings. Partial credit only after required checks pass. Private test split. Expert May 2026 65.4% In use
Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. Expert May 2026 74.2% Not in cards
Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. Expert May 2026 70.7% Not in cards
ViBench 2.0 Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans. Expert May 2026 64.7% Not in cards
VitaBench 2.0 Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics. Everyday May 2026 50.3% In use
WebHarbor Agents navigate local copies of 153 websites to search, sign in, book services, and change stored state. The suite contains 3,518 tasks, including 306 verified test tasks and 2,288 training tasks. The test score measures the percentage of tasks passed by scripted verifiers. Everyday May 2026 86.7% Not in cards
Workspace-Bench-Lite Agents complete 100 workspace tasks. The leaderboard reports the percentage of 1,856 rubric checks passed, with separate runs for each model and harness. University May 2026 66.7% Not in cards
WorkSpaceBench Agent completes 388 tasks in realistic worker file workspaces (20k files) needing cross-file dependency reasoning. Scored against 7,399 rubrics. University May 2026 67.7% In use
WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. May 2026 69.5% Not in cards
Zerolang Editing Agents repair and refactor Zerolang programs through graph editing tools. The evaluation includes 67 held-out synthetic tasks. University May 2026 Not in cards
τc-Bench Agents use tools while conversing with simulated customers in airline, retail, and telecom tasks. The evaluator compares the final environment state with the expected state. Generated scenarios add unusual tool combinations, ambiguous requests, and distracting database records. Everyday May 2026 87.7% Not in cards
AARRI-Bench Agents complete research intern tasks that test scientific judgment, professional conduct, and attention to detail. The Harbor collection contains 82 tasks. Expert Apr 2026 68.3% Not in cards
Agent Bazaar Agents price goods or trade with sellers in simulated markets. The benchmark measures whether their decisions sustain prices, deter fraud, and preserve welfare and profits. University Apr 2026 Not in cards
AlphaEval Agent products such as Claude Code and Codex do 94 production tasks from 7 companies across 6 O*NET domains. Grading mixes formal checks, UI tests and rubrics. Expert Apr 2026 64.4% Not in cards
AlphaZero Connect Four study Coding agents implement an AlphaZero pipeline for Connect Four. The resulting programs play a round-robin tournament against the Pascal Pons solver. Expert Apr 2026 1,938 Not in cards
AutomationBench Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. University Apr 2026 56.3% In use
AutoResearchBench Agents search the scientific literature, either tracking down one target paper through multi-step probing or collecting every paper that meets given conditions. Expert Apr 2026 9.39% Not in cards
BankerToolBench 100 end-to-end junior-banker tasks producing Excel, PowerPoint and Word deliverables. Agentic grader scores ~150 weighted expert rubric criteria. Expert Apr 2026 16% In use
BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. Expert Apr 2026 83.6% System cards only
BRIGHT-Pro Models retrieve passages that cover different parts of a question. The 739 questions include 115 from Earth Science Stack Exchange. University Apr 2026 41.8 Not in cards
BTF-2 Agents forecast 1,417 binary questions asked in October 2025 and resolved in December 2025, researching offline in a frozen corpus of 15 million documents. Scored by Brier score. Expert Apr 2026 0.13 Not in cards
Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. Expert Apr 2026 97.1% Fading
Clash of AIs Season 1 Models publish trade entries, targets, and stops before prices move. Clash of AIs simulates fills against market prices and ranks accounts by total equity. University Apr 2026 $283,903 Not in cards
Claw-Eval Pass^3 Agents complete 300 human-verified tasks in service orchestration, multimodal interaction, and professional dialogue. Pass^3 credits a task only if all three trials pass. University Apr 2026 78.3% In use
Claw-Eval-Live Agents complete 105 workflow tasks in 17 families drawn from live ClawHub workflow signals, scored on both the workflow and the final workspace state. University Apr 2026 66.7% Not in cards
Claw-Eval-MM Agents complete the 101 multimodal tasks of Claw-Eval, such as video questions, document extraction, and webpage generation. Rubric items graded from run traces set scores. University Apr 2026 54.7% In use
ClawBench Browser agents do 153 everyday tasks on live websites, such as booking travel, ordering food and applying for jobs. Request interception stops real side effects, and an LLM judge scores the five-layer traces. Everyday Apr 2026 61.4% Not in cards
CocoaBench Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output. Expert Apr 2026 45.1% Not in cards
CodeIPI Coding agents fix bugs while handling untrusted repository text. The benchmark scores task completion, resistance to embedded instructions and false alarms on clean tasks. University Apr 2026 91.1% Not in cards
Cooperate to Compete (C2C) Multi-player conquest game with private non-binding negotiation and secret asymmetric objectives. LM agents and humans are ranked by Plackett-Luce player strength from their game results. Everyday Apr 2026 0.74 Not in cards
CUA-World-Long Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task. Expert Apr 2026 27.5% Not in cards
DataSciBench Agents complete open-ended data-science tasks that require data cleaning, analysis, visualization and modeling. Component checks and overall task success score the work. Expert Apr 2026 Not in cards
DeepSeek Chinese white-collar tasks evaluation Agents complete 30 Chinese professional analysis, document generation and editing tasks across 13 industries. Blind human annotators compare outputs against a baseline model. Expert Apr 2026 53% System cards only
DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task. Expert Apr 2026 67% System cards only
DeepSeek search Q&A evaluation Models answer objective and subjective Chinese search questions with retrieval or agentic search. Pairwise judging gives win, loss and tie rates. Everyday Apr 2026 28.1% System cards only
DELEGATE-52 Models perform long delegated document-editing workflows across 52 professional domains. Scores measure artifact fidelity after repeated transformations. Expert Apr 2026 Not in cards
Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. Expert Apr 2026 83% Fading
FrogsGame post-training task Agent has 20 hours to post-train Qwen3-8B to solve FrogsGame puzzle boards. A hidden verifier scores the best checkpoint on unseen boards. Research frontier Apr 2026 34% Not in cards
FrontierSWE v1 Agents work up to 20 hours on 17 ultra-long engineering, performance and research tasks in their own harnesses. The board ranks models by average task rank. The score here is dominance, the win rate against a random baseline, which labs report. Research frontier Apr 2026 88.2% In use
Full-Duplex-Bench v3 Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking. Everyday Apr 2026 68% Fading
Gaia2-CLI Agents complete realistic tasks through command-line tools. The benchmark measures task success across search, execution, adaptability, ambiguity, and time constraints. Everyday Apr 2026 57% Not in cards
Game Arena Five in a Row Language models play a gomoku-style game and try to line up five of their tokens before the opponent does. Everyday Apr 2026 1,131 Not in cards
GameWorld Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. Everyday Apr 2026 76.8% System cards only
GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. Expert Apr 2026 33.2% Fading
GeoNatureAgent Benchmark Agents use a geospatial API to analyze environmental indicators in Spain and Portugal. The grader checks tool use and answers against task requirements. University Apr 2026 60.8% Not in cards
GomokuBench Models place stones on a 19 by 19 board against a search engine. The score rewards wins, quicker wins, and longer losses. Everyday Apr 2026 2.90 Not in cards
GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. Expert Apr 2026 11.4% Not in cards
Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. Research frontier Apr 2026 7.60% System cards only
Hiding a secret password (Anthropic) Models keep a given password secret while an investigator model tries to extract it over many turns. Measures survival and per-turn leak rates. Expert Apr 2026 System cards only
HiL-Bench Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary. Expert Apr 2026 79.7% Not in cards
HWE-Bench v1.0 (RTL bug repair) Agents fix bugs in six open-source hardware projects. The benchmark reports the share of fixes that pass repository tests in simulation. Expert Apr 2026 79.9% Not in cards
Hy-Backend Coding agents complete Tencent in-house backend development tasks. Task and scoring details are not published. Expert Apr 2026 54.7% Fading
Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task. Expert Apr 2026 64.2% In use
Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published. Expert Apr 2026 50.5% Fading
ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. Expert Apr 2026 30.7% Fading
Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. Research frontier Apr 2026 81.8% System cards only
KellyBench Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth. Expert Apr 2026 -7.90% Not in cards
Keystone-Eval Agents configure Dockerfiles and test entrypoints for arbitrary code repositories, then run the repositories' tests. Expert Apr 2026 Not in cards
KnowU-Bench Agents use Android apps such as Mattermost and Taodian. They infer preferences, ask for missing details, and decide when to act or stay silent. Everyday Apr 2026 60.4% Not in cards
LinuxArena AI-control setting in live Linux production-like environments. Measures stealthy side-task completion against action and CoT monitors. Expert Apr 2026 0% System cards only
Malicious agentic influence campaigns (Anthropic) Agents receive requests to run multi-step influence operations with tools. Score is the refusal rate. Everyday Apr 2026 System cards only
Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. Expert Apr 2026 89.9% System cards only
MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE. Expert Apr 2026 77.1% Fading
Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. Expert Apr 2026 2.60% System cards only
MMSearch Multimodal agents search and answer questions that require visual and textual evidence. Expert Apr 2026 72.9% Fading
neuralCAD-Edit Agents edit existing CAD models from text or recorded instructions. CAD experts rate whether they would accept each edit. Software also compares the resulting geometry. University Apr 2026 0.25 Not in cards
o11y-bench Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment. Expert Apr 2026 93.8% Not in cards
OccuBench Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable. Expert Apr 2026 79.6% Not in cards
OpenGameEval Expanded Eval Set Agents extend existing Roblox games and build features from empty scenes. Roblox added 40 coding tasks to the original 47. University Apr 2026 50.3% Not in cards
Pare-Bench Proactive assistants infer user goals and act across stateful digital applications with simulated users and multi-app task environments. Expert Apr 2026 Not in cards
PHMForge Agents analyze battery and aircraft engine data to diagnose faults, estimate remaining life, and make maintenance decisions. Harbor packages 99 industrial scenarios as 123 tasks. Expert Apr 2026 80.8% Not in cards
QuantSightBench Models give 90% prediction intervals for 1,000 numerical forecasting questions with known outcomes. The main setting allows retrieval from a fixed news corpus. Mean Log Interval Score penalizes wide intervals and missed outcomes, while coverage measures how often intervals contain the answer. Expert Apr 2026 6.70 Not in cards
QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. Everyday Apr 2026 60.5% Fading
RECON Research agents find and verify several facts about identified people on the open web without mixing up people with similar names. Expert Apr 2026 36.2% Not in cards
Riemann-bench 25 private research-mathematics problems from professors and IMO medalists. Agents with tools, 100 runs per problem. Research frontier Apr 2026 83.2% In use
RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. Everyday Apr 2026 46.2% In use
Spatial-Gym Models find paths that satisfy grid puzzle rules. The evaluation uses 500 SPaRC puzzles in one-shot and stepwise settings, including a setting that allows backtracking. School Apr 2026 Not in cards
SpireBench (HermesBridge trial-v0) Models play Slay the Spire 2 from structured game states. Each plays all five characters at the base difficulty, with one run per character. Everyday Apr 2026 28.0 Not in cards
Supabase Evals Agents build, deploy, investigate, and repair Supabase projects. The hub port uses OpenCode and runs the original task scorers from a pinned release. Expert Apr 2026 92.8% Not in cards
SWE Atlas Test Writing Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests. Expert Apr 2026 67% In use
SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. Expert Apr 2026 42.9% System cards only
TarantuBench Agents complete simulated web security challenges. Each challenge has a binary success criterion. Expert Apr 2026 Not in cards
The Amazing Agent Race Agents navigate websites, call tools, and calculate a passcode across 1,400 scavenger hunts. The benchmark includes 800 sequential puzzles and 600 with branching paths. Expert Apr 2026 36.6% Not in cards
Toloka Arena Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed. Expert Apr 2026 33% Not in cards
UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. Research frontier Apr 2026 80% System cards only
UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. Expert Apr 2026 95% System cards only
UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. Expert Apr 2026 0% System cards only
UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. Expert Apr 2026 0% System cards only
VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. Everyday Apr 2026 43.5% Not in cards
Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. Expert Apr 2026 System cards only
VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. Expert Apr 2026 Not in cards
VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. Research frontier Apr 2026 System cards only
AEC-Bench (Nomic) Agents search construction drawings and specifications to answer questions and identify conflicts. Task verifiers compare their outputs with known findings and award partial credit. University Mar 2026 70.2 Not in cards
AndroidBench Android development tasks from popular open-source apps: agent resolves issues or builds features, verified by test suites in emulator environments. Expert Mar 2026 91.8% In use
ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. Everyday Mar 2026 100% In use
AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. Expert Mar 2026 76.5% Not in cards
ATM-Bench Agents answer questions about a person’s history using images, videos, and email records. The Harbor package contains 31 hard questions with schema-guided memory. University Mar 2026 86% Not in cards
AutoResearch An agent edits a small nanochat training script and runs 5 minute training experiments overnight. Scored by final validation bits per byte, where lower is better. Research frontier Mar 2026 Not in cards
BarterBench Agents trade scarce goods to meet inventory targets. The hub port assigns one model-controlled trader to a marketplace of scripted opponents. University Mar 2026 Not in cards
BIBIMBAP Agents read and edit IFC building files. Executable tests compare their changes and answers with reference results. University Mar 2026 50.2% Not in cards
BigLaw Bench: Research Agents use search tools to find US case law and give cited answers. The tasks span corporate law, litigation and other practice areas. Expert Mar 2026 Not in cards
BixBench-Hypothesis Agents analyze existing datasets to test 51 hypotheses. A rubric awards points for intermediate results and the final conclusion. Expert Mar 2026 Not in cards
BrokenArXiv Models are asked to prove statements from recent arXiv papers that were changed to be plausible but false. Full credit goes to models that say the statement is false. Research frontier Mar 2026 84.4% In use
CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. University Mar 2026 32.3% Not in cards
Claw-Eval 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Everyday Mar 2026 81.4% In use
Clocktower Radio Models play the social deduction game Blood on the Clocktower (the Trouble Brewing script) in mirrored pairs of games. In each match one model plays good and the other evil, and then they swap. A coded Storyteller runs the game, and a Bradley–Terry fit over all matches gives ratings where 1500 is average. Everyday Mar 2026 1,708 Not in cards
Cooperative Codenames (Max Bittker) Two models cooperate as cluegiver and guesser to identify target words on a hidden board. The default evaluation uses 200 separately generated boards. Everyday Mar 2026 Not in cards
Crate Bench Models identify an album and its artist from the cover image. The benchmark provides 300 test covers and supports closed-book and web-search modes. Everyday Mar 2026 Not in cards
CTI-REALM Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry. Expert Mar 2026 63.7% Not in cards
CursorBench Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.0 covers edit, refactor and bugfix problems. Agentic graders score correctness, with cost reported. Expert Mar 2026 61.3% System cards only
Data Agent Benchmark Agents answer business questions over data split across several database systems, with duplicated and inconsistent records and key facts buried in text. The 104 queries cover 17 datasets and four database systems, based on a study of companies building data agents in six industries. Scores are pass@1 against known answers. Expert Mar 2026 70.9% Not in cards
Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. Expert Mar 2026 System cards only
EnterpriseOps-Gym Agents do 1,150 expert-written tasks in eight enterprise domains, such as IT, HR and customer service, using 512 tools over live MCP servers. SQL checks on the final database state score each task. Expert Mar 2026 45.9% In use
FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. University Mar 2026 81% Not in cards
FinToolBench Agents answer 295 finance questions by selecting and executing tools from a library of 760 financial interfaces. The evaluation separates tool execution, answer correctness, and mismatches in timeliness, intent, and regulatory domain. Answer checks combine model judging with exact matching. Expert Mar 2026 0.82 Not in cards
FLTEval Agents complete the formal Lean proofs and define the new mathematical concepts in real pull requests to the Fermat's Last Theorem (FLT) formalization project. Lean and SafeVerify check each submitted diff. Research frontier Mar 2026 28.9 In use
Gray Swan IPI Arena benchmark Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Mar 2026 0% In use
GTO Wizard Benchmark Models play heads-up no-limit Texas Hold’em against GTO Wizard AI. AIVAT reduces variance while the benchmark reports poker winnings. Expert Mar 2026 -9.23 Not in cards
GUI-CEval Models answer questions about Chinese app screens, locate controls, and choose actions. Agents also complete tasks on physical devices, where the grader checks interface elements. Everyday Mar 2026 61.4% Not in cards
InteractiveBench Texas Hold’em Models play no-limit Texas Hold’em from partial observations of cards, chip stacks, and recent actions. Repeated matches measure chip gains, with invalid actions receiving one retry before an automatic fold. Competition Mar 2026 31.8 Not in cards
InteractiveBench Trust Game Models choose whether to cooperate or defect in repeated prisoner’s dilemma matches. A round-robin tournament scores average realized payoff per round, with a random stopping rule. Everyday Mar 2026 1.87 Not in cards
Kaggle Standardized Agent Exams Agents answer 16 questions on multistep reasoning and handling manipulative prompts. Kaggle publishes an overall score. School Mar 2026 Not in cards
KARLBench Agents answer questions using vector search over fixed document collections. The suite measures six kinds of evidence gathering and reasoning. University Mar 2026 67.5 Not in cards
LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. Everyday Mar 2026 71.4% Not in cards
LiveClawBench Agents act as OpenClaw personal assistants on 134 multi-step tasks across 10 app domains backed by stateful mock services. A run passes when its reward exceeds 0.8. Expert Mar 2026 76% In use
MADQA Agents search 800 PDF documents to answer 2,250 human-written questions. Scoring covers answer accuracy and how much search effort the agent spends. Everyday Mar 2026 85.2% In use
ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. Everyday Mar 2026 Not in cards
MM-ClawBench Agents complete common OpenClaw tasks such as study planning, office documents, scheduled research, and code maintenance. Scoring is accuracy. Everyday Mar 2026 75.4% Fading
Monthly-SWEBench Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1. Expert Mar 2026 76% Not in cards
MyClaw Bench Models run inside the OpenClaw agent on 45 tasks in four difficulty tiers built from real usage data. Some tasks use an LLM judge. Everyday Mar 2026 Not in cards
NYT Connections (Prime Community) Models identify four groups of related words in New York Times Connections puzzles. The default 100-puzzle evaluation scores completion, efficiency, partial progress, and format. Everyday Mar 2026 Not in cards
OfficeQA Pro Models answer the 133 hardest OfficeQA questions by finding tables in U.S. Treasury Bulletin documents and computing exact numeric answers. University Mar 2026 60.9% In use
OpenGameEval Debug Evals Agents diagnose and fix faults in Roblox Studio games. The authors created 30 debugging tasks from 15 scenarios. University Mar 2026 64.7% Not in cards
PE Civil Bench Models solve 110 multiple-choice and 40 numerical civil engineering exam problems. A model compares each answer with the reference solution. University Mar 2026 87.3% Not in cards
Pencil Puzzle Bench Models solve 300 pencil puzzles of 20 varieties with unique solutions, either in one shot or agentically with step checks. Every board state can be verified against the rules. Expert Mar 2026 97.6% Not in cards
PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. Everyday Mar 2026 75.1% Not in cards
PokeAgent Challenge Agents compete in Pokemon battles under partial observability in a Battling Track and play the Pokemon RPG in a Speedrunning Track that needs long-horizon planning. Expert Mar 2026 Not in cards
PostTrainBench CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Research frontier Mar 2026 34.1% In use
RetailBench Agents operate a simulated supermarket for up to 180 days. They choose suppliers, order stock, and set prices as goods age and customers respond. University Mar 2026 $24,351 Not in cards
RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. Everyday Mar 2026 10.4% Not in cards
RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. Everyday Mar 2026 23.9% Not in cards
RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. Everyday Mar 2026 0% Not in cards
RVMS-Bench Agents search open-domain videos and localize moments using global impression, key moment, temporal context and auditory memory cues. Expert Mar 2026 Not in cards
SlopCodeBench Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity. Expert Mar 2026 32.1% Not in cards
SWE Atlas Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. Expert Mar 2026 63.2% In use
SWE-CI Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes. Expert Mar 2026 92% Not in cards
tau3-bench Agents use tools and follow policies while talking with simulated users in airline, retail, telecom, and banking domains. Scoring is task success rate. Everyday Mar 2026 96% In use
VAKRA Agents answer multi-step questions by calling more than 8,000 locally hosted APIs over real databases in 62 domains, along with matching document collections. Tasks need chains of 3 to 7 steps across APIs and documents. A deterministic evaluator replays tools and checks the full trajectory. Everyday Mar 2026 38.1% Not in cards
Vibe Code Bench End-to-end web application development from natural-language specs. Built apps are tested against functional requirements. Expert Mar 2026 93.8% In use
Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Expert Mar 2026 82.3% In use
Vision2Web S1-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Expert Mar 2026 52.7% Not in cards
Vision2Web S2-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Expert Mar 2026 43.5% Not in cards
WildClawBench 60 original in-the-wild tasks run inside a live OpenClaw assistant environment, chaining 10-60+ tool calls. Scored by task-level graders. University Mar 2026 67.2% In use
ZClawBench Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use. Expert Mar 2026 57.6% Fading
τ-Voice Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction. Everyday Mar 2026 81.7% In use
τ³-bench Banking Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Everyday Mar 2026 55.1% In use
AD-Bench Agents answer advertising analytics requests with production-platform tools. A model judge checks final answers against refreshed reference answers and measures coverage of expert tool-call trajectories. Results report success over one or three attempts. University Feb 2026 83% Not in cards
Agent-Diff Agents use code to complete tasks across replicas of Slack, Linear, Box, and Calendar. The evaluator compares the resulting application state with the expected changes. University Feb 2026 88.1% Not in cards
Agentic code behavior scores (Anthropic) Agents work through over one hundred realistic coding scenarios. Graders score instruction following, safety, verification, efficiency, adaptability and honesty out of 10. Expert Feb 2026 System cards only
AI GameStore Vision-language models play short episodes of 100 games adapted from Apple App Store and Steam top charts. Each game score is divided by the average human score, and models are ranked by the geometric mean. Everyday Feb 2026 12.4% Not in cards
AIDABench Agents analyze documents and spreadsheets to answer questions, draw charts, and generate files across 603 tasks. They can execute Python in a sandbox. Separate model evaluators check answer correctness, chart correctness and readability, and spreadsheet structure and content. Expert Feb 2026 59.4% Not in cards
AIRS-Bench Agents carry out 20 AI research tasks taken from recent ML papers, without baseline code. Scores are normalized so that 1.0 matches the published human state of the art. Research frontier Feb 2026 0.40 System cards only
ArXivMath Final-answer research-level math problems extracted monthly from recent arXiv papers and filtered to be self-contained. Research frontier Feb 2026 94.3% In use
Audio Arena Speech-to-speech models hold one long spoken conversation per scenario and call tools when needed, such as booking a dental appointment or ordering groceries. Six scenarios add up to 221 spoken turns with corrections, false memory traps, prompt injection and state tracking. An LLM judge grades each turn against a golden answer. Everyday Feb 2026 85.7 Not in cards
AutoElicit An optimizer perturbs benign instructions to elicit unsafe behavior from computer-use agents. Reports elicitation success rate on transferred prompts. Expert Feb 2026 37.6% System cards only
Bio-Literature Search Level 2 Agents use search tools to compare abstracts from PubMedQA, including their sections, subject headings, and conclusions. Five task types require retrieving or combining evidence. University Feb 2026 Not in cards
BioMysteryBench Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. Expert Feb 2026 83.7% In use
BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. Expert Feb 2026 87.7% System cards only
Blicket Test Causal Reasoning Models experiment with a simulated detector to infer which objects activate it. The hub version rewards correct identification and informative experiments. Everyday Feb 2026 0.56 Not in cards
CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. Everyday Feb 2026 Not in cards
CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. Expert Feb 2026 25.8% Fading
CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. Expert Feb 2026 74.8% Fading
CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. Expert Feb 2026 72.2% Fading
CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. Expert Feb 2026 52.3% System cards only
ContextBench Coding agents resolve 1,136 issues from 66 repositories in eight programming languages, and the benchmark checks which code they look at along the way. Each task has human-annotated gold context, and the tracker scores context recall, precision and efficiency over the agent's trajectory. It adds these process measures to the usual issue-resolution result. Expert Feb 2026 60.6% Not in cards
Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. Expert Feb 2026 16.4 System cards only
DARE-Bench Agents write code to produce predictions from structured files. The benchmark checks predictions and adherence to modeling instructions. University Feb 2026 49.3% Not in cards
Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. Expert Feb 2026 39.6% System cards only
DEEPSYNTH Agents search multiple websites and combine evidence to answer 120 research tasks across seven domains. They return structured JSON answers scored with precision, recall, F1, and exact match. A separate model judge assesses answers beyond exact matching. Expert Feb 2026 8.97% Not in cards
DISBench Multimodal agents search raw visual histories over several steps to find target images from implicit context cues. Everyday Feb 2026 28.7% Not in cards
DRACO 100 deep-research tasks from real Perplexity queries across 10 domains, graded on about 40 expert rubric criteria each. Expert Feb 2026 88.6% In use
DraftNEPABench Agents draft sections of environmental impact statements using technical and regulatory source documents. Model judges score the drafts against rubrics, and specialists assess a sample. Expert Feb 2026 2.41 Not in cards
EnterpriseBench: CoreCraft Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy. Expert Feb 2026 77.4% In use
EnterpriseRAG-Bench Retrieval systems and agents answer 500 questions over a synthetic company's internal documents, slightly over 500,000 of them. Expert Feb 2026 91.5% Not in cards
EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. Expert Feb 2026 51% Not in cards
FeatureBench Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically. Expert Feb 2026 60.2% Not in cards
FeatureBench Lite The public 30-task Lite leaderboard ranks passed-test percentages and reports fully resolved-task percentages separately. Expert Feb 2026 60.2% Not in cards
Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. Expert Feb 2026 21.6% Fading
FPL 2026 Agentic FPGA Backend Optimization Competition Teams improve the maximum clock frequency of routed FPGA designs while preserving behavior. The organizers rank teams across seven designs after accounting for runtime and model cost. Expert Feb 2026 Not in cards
Game Arena Chess Openings Language models play chess against each other from varied two-move openings sampled from 20 popular human openings. The board is given as text. Expert Feb 2026 1,263 Not in cards
Game Arena Four in a Row Language models play a Connect Four style game and try to line up four of their tokens before the opponent does. Everyday Feb 2026 1,003 Not in cards
Game Arena Poker Language models play heads-up no-limit Texas Hold'em against each other. Models are ranked by mean big blinds won per 100 hands. Expert Feb 2026 30.3 Not in cards
Game Arena Werewolf Teams of language models play the social deduction game Werewolf in natural language. Villagers must find the hidden werewolves, who try to deceive them. Expert Feb 2026 0 Not in cards
GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result. University Feb 2026 70.6% Not in cards
GDPval-MM Agents produce office deliverables for GDPval tasks in a MiniMax agent framework. A judge model compares full trajectories pairwise and reports a win rate. Expert Feb 2026 73.5% Fading
IFC-Bench v2 Agents inspect IFC files and answer questions about building elements and their relationships. A model grades each answer against a reference. University Feb 2026 65.8% Not in cards
LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. Expert Feb 2026 60.9% In use
LABBench2 LitQA3 Models answer open-response questions whose answers appear once in recent biology papers. The LitQA3 subtask of LABBench2. Expert Feb 2026 85.1% Not in cards
LABBench2 PatentQA Models locate and interpret evidence in patent documents. Expert Feb 2026 90.9% Not in cards
LHAW Agents face 285 underspecified variants of long-horizon tasks with goals, constraints, inputs or context removed. Scored on strategic clarification and task success. Expert Feb 2026 68.4% Not in cards
LLM-WikiRace Models play the Wikipedia game: starting from one article, they follow one hyperlink at a time to reach a target article. At each step the model sees only the links on the current page, and a game is lost after 30 steps. Three difficulty levels are set by the shortest path between the two pages. Scores are the share of games won at each level. Everyday Feb 2026 23% Not in cards
LOCA-Bench Agentic tasks whose environment state is scaled to grow context length while task semantics stay fixed. Success rate versus context length. University Feb 2026 21.3% In use
MemGUI-Bench Agents remember details while working across Android apps. Tasks include recording audio and creating calendar events. A model checks the actions and results across repeated attempts. Everyday Feb 2026 73.4% Not in cards
MEWC Models solve 179 Microsoft Excel World Championship problems from 2021 to 2026 with Excel tools. Output cells are compared one by one with answer cells. Competition Feb 2026 74.4% Fading
MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. Everyday Feb 2026 37.7% Fading
Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. Research frontier Feb 2026 76% System cards only
OdysseyArena Agents explore interactive environments built from four transition primitives and must work out the hidden dynamics over long horizons. OdysseyArena-Lite has 120 standard tasks. Expert Feb 2026 44.2% Not in cards
OdysseyArena AI Trading Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials. Expert Feb 2026 23.3% Not in cards
OdysseyArena Energy Dispatch Agents manage energy dispatch under hidden rules. The score is average task success over four trials. Expert Feb 2026 23.3% Not in cards
OdysseyArena Repo System Agents operate repositories with changing hidden rules. The score is average task success over four trials. Expert Feb 2026 62.5% Not in cards
OdysseyArena Turn on Lights Agents learn hidden switches and turn on lights. The score is average task success over four trials. Expert Feb 2026 28.3% Not in cards
OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. University Feb 2026 62.5% Fading
OpenEarthAgent Agents use tools such as QGIS to measure features in satellite images and maps. The authors grade tool decisions and final answers. University Feb 2026 44.5% Not in cards
OpenThoughts-TBLite Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates. Expert Feb 2026 75.1% Not in cards
Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. Expert Feb 2026 9.40% System cards only
PinchBench Real-world tasks (scheduling, email triage, coding, research, files) for an LLM driving an OpenClaw agent. Graded by automated checks and LLM judge. Everyday Feb 2026 94.9% In use
ProxyWar Models write bots for nine games. The harness tests each bot and runs tournaments to measure wins, errors, and decision time. University Feb 2026 39.6% Not in cards
Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. Expert Feb 2026 70% System cards only
RISE Agents answer real questions from human experts that need multi-step search and interaction with information-dense web pages. MiniMax has not published the scoring method. Expert Feb 2026 64.3% Fading
RuneBench Coding agents play RuneScape on an emulated game server by writing TypeScript against the rs-sdk library, with game-wiki notes for reference. Tasks train one skill or gather gold within 15 or 30 minutes. Skill tasks are scored by the peak XP rate in any 15-second window, and a verifier reads the final game state. Everyday Feb 2026 7.85 Not in cards
SalesBench (Insurance Sales) Agents find insurance leads, quote plans, and schedule callbacks in a simulated sales pipeline. The evaluator rewards converted monthly premiums and checks compliance with calling restrictions. University Feb 2026 Not in cards
scBench (LatchBio SingleCellBench) 195 verifiable single-cell RNA-seq analysis problems. Each pairs a data snapshot with a task and a deterministic grader. Expert Feb 2026 63.9% System cards only
scBench (original study) Agents complete 394 tasks in single-cell data analysis. The authors compare their answers with results from the original analyses. Expert Feb 2026 52.8% Not in cards
SDR-Bench Models personalize sales pitches using customer research. A judge compares each pitch with points from successful human outreach. University Feb 2026 55.8% Not in cards
SimulCost Agents tune physics simulations to meet an accuracy target. The authors measure success and computational cost for an initial guess and for repeated attempts with feedback. Expert Feb 2026 80.8% Not in cards
SkillInject Agents use a skill file that carries an adversarial instruction while doing a legitimate task. Reports how often the agent follows the injection. Expert Feb 2026 0.60% System cards only
SkillsBench 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. Expert Feb 2026 66.5% In use
SOP-Bench Agents follow expert-written standard operating procedures across 12 industrial domains, using executable tool interfaces over more than 2,000 tasks with 10 to 50 decision points. Expert Feb 2026 Not in cards
Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. Expert Feb 2026 79% System cards only
Swarm Bench Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. Expert Feb 2026 76.3% System cards only
SWE-rebench V2 Agents repair real repository issues using executable tests. The authors compare models on 300 tasks across five languages. Harbor also ports 51 validated Clojure tasks. Expert Feb 2026 25.2% Not in cards
Terminal-Bench 2.0 Verified (Z.ai) Agents do the Terminal-Bench 2.0 tasks after Z.ai fixed environments for 89 tasks and instructions for 11. A further 2026-08-18 round fixed tests and instructions in 6 more tasks. Expert Feb 2026 61.8% Not in cards
Vals Finance Agent Agents answer 537 financial analyst questions on public company SEC filings with search and filing tools. A three-vote GPT-5.2 judge grades each final answer. Expert Feb 2026 64.4% Fading
VDR-Bench Models answer 2,000 visual questions that require finding entities in images and researching related facts. A Qwen3-VL judge checks final answers against references. Entity recall separately measures whether searches retrieve the relevant entities. University Feb 2026 27.2% Not in cards
VenusBench-Mobile Agents use Android apps such as VLC, Markor, and Pro Expense. A model judges 90 primary tasks. The other tasks use programmatic checks. Everyday Feb 2026 24.8% Not in cards
VeRO-Bench Coding agents improve other agents by editing their prompts, tools, and workflows. Each task has a reference evaluation for comparing changes. Expert Feb 2026 0.52 Not in cards
VIBE-Pro (MiniMax) Agents build full-stack projects for Web, Android, iOS and Windows, including server APIs and databases. Claude Code based verifiers check interaction logic and visual output. Expert Feb 2026 55.6% Fading
Wiki Live Challenge Deep research agents write articles on topics of 100 recent Wikipedia Good Articles, which serve as expert references. Scored on 39 writing criteria and factual verifiability. Expert Feb 2026 53.6 Not in cards
Windsurf Arena Mode Leaderboard Developers compare two model responses to the same coding request inside the editor. Their choices determine model Elo ratings. University Feb 2026 1,098 Not in cards
WordleArenas Models guess a five-letter word from colored clues. Each arena changes how many clues give false feedback after a guess, from zero to five. Everyday Feb 2026 100% Not in cards
YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. University Feb 2026 $3,445,495 In use
ABC-Bench Coding agents take a real backend repository, edit the code, set up the environment and deploy the service in a container. The 224 tasks come from 127 open-source repositories in 8 languages and 19 frameworks. A task passes only when the running service passes external end-to-end API tests. Expert Jan 2026 63.2% Not in cards
AgencyBench Agents complete 138 long tasks drawn from daily AI use, grouped into 32 scenarios that cover six agentic capabilities such as coding, front-end work, game development and research. A typical task takes about 90 tool calls, 1 million tokens and hours of work. A simulated user gives feedback during the run, and a Docker sandbox checks the deliverables against task rubrics. Expert Jan 2026 56.5% Not in cards
Agent Poker Bench Models play 10-seat no-limit Texas Hold'em tournaments against each other. Chip counts and elimination order feed TrueSkill ratings. Expert Jan 2026 1,132 Not in cards
AI Office Benchmark (Moonshot) Agents complete end-to-end office tasks and produce documents, spreadsheets, PDFs and slide decks. Experts rate output quality. Expert Jan 2026 Not in cards
APEX-Agents 480 long-horizon tasks in banking, consulting and law worlds with docs, sheets, email and chat. Rubric-graded, Pass@1. Expert Jan 2026 67.2% In use
APEX-SWE Software-engineering productivity tasks spanning integration work and observability, such as diagnosing failures from telemetry. Expert Jan 2026 67.6% In use
Artificial Analysis Intelligence Index v4.0 Version 4.0 of Artificial Analysis's index, ten evaluations in four equal categories that brought in real-world knowledge work and hallucination tests. Jan 2026 60.2 Fading
Bash Agent Evals Agents query GitHub issues and pull requests through bash, SQL, or a combination. Braintrust scores answers against the dataset. University Jan 2026 Not in cards
BU Bench V2 Browser agents do real web tasks with the Browser Use library. A judge scores each run against a rubric. Everyday Jan 2026 80.6 Not in cards
Buyout Game Eight language models play a money-driven elimination game with private transfers and a buyout endgame. Everyday Jan 2026 1975.4 Not in cards
CAR-bench Agents act as an in-car voice assistant that talks with an LLM-simulated driver and uses 58 connected tools for navigation, productivity, charging and vehicle control under set policies. Besides ordinary tasks, hallucination tasks remove a needed tool or fact, and disambiguation tasks need a clarifying question or a lookup. Scores reward consistent success across repeated runs. Everyday Jan 2026 70% Not in cards
CooperBench Two coding agents each get a different feature for the same repository and must coordinate so both changes work together. Over 600 tasks span 12 libraries in 4 languages. Expert Jan 2026 27.9% Not in cards
DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. Everyday Jan 2026 58.9% Fading
DeepResearch Bench II Deep research agents write reports for 132 research briefs in 22 domains, each derived from a review article they may not consult. A judge model checks 9,287 expert criteria. Expert Jan 2026 43.1% Not in cards
DeepResearch Bench II (Gemini 2.5 Pro judge) Systems conduct deep research under the legacy Gemini 2.5 Pro evaluator. Scores remain separate from the current GPT-5.5 evaluator. Expert Jan 2026 45.4% Not in cards
DeepResearchEval Deep research systems answer complex research tasks that require evidence from many web sources. A persona-driven pipeline writes the tasks and keeps only those that need outside search. A model judge scores each report on general and task-specific dimensions, and an agent fact-checks the report's statements with web search even when citations are missing. Expert Jan 2026 Not in cards
DeepResearchEval v1 Research agents prepare reports. The score is average report quality on the original evaluation tasks. Expert Jan 2026 Not in cards
DeepSearchQA Questions that need multi-step search and synthesis. Final answers graded against references. Expert Jan 2026 95% In use
DevOps-Gym Agents build, monitor, repair and test Java and Go projects. The Prime port converts tasks to single-turn questions with proxy grading. University Jan 2026 27.5% Not in cards
General Agent Benchmark (Moonshot) Agents complete multi-step production workflows. Experts compare their work with human expert performance. Expert Jan 2026 Not in cards
KernelBench-Verified Language models generate CUDA kernels that must match PyTorch outputs and improve runtime. The verifier checks correctness with robust performance baselines. Research frontier Jan 2026 Not in cards
KradleVerse Agent harnesses compete in a live Minecraft arena. Everyday Jan 2026 Not in cards
LiveNewsBench Release 2 Models search for answers to questions about recent news. The test limits searches and page visits and compares answers with curated references. Everyday Jan 2026 84.5% Not in cards
mage-bench Models play Magic: The Gathering in a benchmark that compares strategic play across games. Expert Jan 2026 1617.0 Not in cards
MegaGem Three models bid in sealed auctions with private cards. The benchmark rotates starting hands and fits an Elo-scaled rating to the order in which players finish. Everyday Jan 2026 Not in cards
METR Time Horizon 1.1 Agents attempt 228 software and research tasks timed against skilled humans, run in Inspect. The score is the human task length at which the agent succeeds half the time. Research frontier Jan 2026 17.4 h Not in cards
MineBench Models turn a prompt into a Minecraft-style voxel build as raw 3D coordinates or tool calls. Users compare builds blind, and votes become rankings. Everyday Jan 2026 2,169 Not in cards
MK-Bench Agents schedule jobs when machines break down or orders arrive or disappear. The grader compares each completion time with the best known schedule. University Jan 2026 6.83% Not in cards
NBABench (Thierry Damiba) Models search NBA player statistics from two seasons and answer 102 generated questions. The reference answers come directly from player records. Everyday Jan 2026 Not in cards
OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported. Expert Jan 2026 81% Not in cards
OTelBench Coding agents add OpenTelemetry tracing, metrics and logging to codebases in several languages. Tests check the instrumentation. Expert Jan 2026 29% Not in cards
OverSearchQA Search-augmented models decide when to abstain from retrieval on unknown, false-premise and underspecified questions. University Jan 2026 Not in cards
PaperScope Agents search multiple scientific papers to compare figures, summarize results, and solve problems. Each question can require evidence from several documents. Expert Jan 2026 41.0 Not in cards
Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. Expert Jan 2026 Not in cards
Prediction Arena Agents research events and trade contracts on Kalshi and Polymarket. The leaderboard ranks accounts by cash plus the market value of open positions. University Jan 2026 $138,945 Not in cards
prinzbench Models research fact patterns under U.S. statutes and regulations and hunt for obscure public information online. 33 private questions run three times give a score out of 99. Expert Jan 2026 91.0 Not in cards
ProofBench Models write Lean 4 proofs of graduate-level theorems, with Mathlib search and code tools for up to 40 turns. Scored by the share that compile. Expert Jan 2026 78% Not in cards
Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. University Jan 2026 35.8% Fading
RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. Everyday Jan 2026 76.3% Not in cards
RWSearch Agents search the live web to answer 200 expert-written Chinese questions that need many retrieval steps, filtering and arithmetic. Each question has one objective answer. Expert Jan 2026 82% Fading
ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. Expert Jan 2026 73.8% Fading
Snorkel Agentic Coding Agents complete 100 multi-step coding tasks in sandboxed containers, spread evenly over four difficulty tiers. Unit tests and rubrics grade both outputs and the agent trajectory. Expert Jan 2026 65.2% Not in cards
SokoBench Models solve Sokoban puzzles that require long-horizon planning and persistent state tracking. Everyday Jan 2026 Not in cards
Spyfall study Models play Spyfall while inferring hidden locations and managing deceptive dialogue. Everyday Jan 2026 1003.0 Not in cards
Strategy Bench Models compete in six social deduction and strategy game environments. Everyday Jan 2026 Not in cards
SWE-gen JS Agents fix issues drawn from 1,000 merged pull requests across 30 JavaScript and TypeScript repositories. Each task includes tests that fail before the reference fix. Expert Jan 2026 Not in cards
tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. Everyday Jan 2026 67.1% Fading
TaxoBench Agents find research papers and organize them into a taxonomy. The evaluator compares their groupings and hierarchy with taxonomies from 72 published surveys. Expert Jan 2026 Not in cards
TetrisBench Models play Tetris head-to-head in the original benchmark. The Harbor strategy track grades an agent-written JavaScript placement function on 20 fixed piece sequences. Everyday Jan 2026 66.7% Not in cards
TradeRank Arena Models choose positions in cryptocurrencies and US stocks using shared market data and simulated capital. The leaderboard ranks portfolios by return. University Jan 2026 0% Not in cards
Video Browser Showdown 2026 marine video search Participants use interactive retrieval systems to find five underwater clips after seeing each target. Each search has a five-minute limit. Everyday Jan 2026 Not in cards
VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. University Jan 2026 76% In use
VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. Everyday Jan 2026 20.5% Fading
Vmax Tasks Agents fix JavaScript bugs from projects including Vue.js, Redux, Docusaurus, and Chalk. Harbor lists 1,043 tasks with bug reports and expected behavior. Expert Jan 2026 Not in cards
AA-Briefcase Artificial Analysis knowledge-work evaluation of professional deliverables. Models ranked by Elo. Expert 2026 1,720 In use
LatchBio Capabilities v1.0 LatchBio cross-benchmark suite of agentic biological data analysis (transcriptomics, epigenomics, variants, pharmacology) with deterministic graders. Expert 2026 58% System cards only
Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. Expert Dec 2025 100% System cards only
AgentIF-OneDay Agents complete 104 tasks that stand for a full day of human work, life and study. Tasks include tedious workflows, inferring rules from examples and changing requirements. Expert Dec 2025 62.6% Not in cards
AI Consumer Index Models with web search handle everyday consumer requests in shopping, food, gaming and DIY. Prompt-specific rubrics with hurdle and grounding criteria grade each answer. Everyday Dec 2025 56.1% Not in cards
AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. Expert Dec 2025 47.7% Fading
AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. Expert Dec 2025 44% Not in cards
AMUSE Multimodal models reason over multi-speaker audio-visual interactions in zero-shot, guided and agentic modes across six task families. Expert Dec 2025 Not in cards
App-Bench Coding agents build six full-stack web applications from a single prompt. Two developers grade implemented features against 151 rubric items. The score uses each tool’s best result from three attempts per application. University Dec 2025 67.5% Not in cards
Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. Expert Dec 2025 90.9% In use
BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. Expert Dec 2025 99.1% System cards only
BinaryAudit Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software. Expert Dec 2025 49.1% Not in cards
CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. Research frontier Dec 2025 8.40 Not in cards
CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. Research frontier Dec 2025 66.5 Not in cards
CNFinBench Models answer financial questions and carry out financial workflows. The benchmark includes tasks on expertise, autonomy, and integrity. Expert Dec 2025 83.7 Not in cards
CubeBench Agents solve Rubik's Cube tasks across symbolic, visual and partially observed settings to measure spatial reasoning and long-horizon state tracking. Expert Dec 2025 Not in cards
CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Expert Dec 2025 100% In use
DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. Expert Dec 2025 47% Not in cards
FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. University Dec 2025 78.3% Fading
FACTS Search Models answer information-seeking questions with a web search tool (the Brave Search API) and must combine what they find correctly. University Dec 2025 85.6% Not in cards
Finch (FinWorkBench) Agent completes 172 enterprise finance and accounting spreadsheet workflows (384 tasks) from real data. Human plus calibrated LLM-judge grading. Expert Dec 2025 43% In use
FrontierCS 1.1 Agents solve 179 algorithmic tasks under revised scoring that rewards solutions beyond the reference. The score is the official average task score. Research frontier Dec 2025 56.5 Not in cards
GDPval-AA Agents complete GDPval occupational tasks. Artificial Analysis compares the outputs in pairs and ranks models by Elo. Expert Dec 2025 1,932 Fading
LocalSearchBench Agents search merchant listings to answer 900 questions about local services across nine Chinese cities. A model judges each answer against a reference. Everyday Dec 2025 35.6% Not in cards
MCP-SafetyBench Agents complete tool-use tasks across five domains. The benchmark reports both task completion and resistance to adversarial inputs. University Dec 2025 21.2% Not in cards
MedCalc-Bench Verified Models extract values from patient notes and compute results for 55 medical calculators. Verified release of MedCalc-Bench with corrected labels. Expert Dec 2025 86.2% System cards only
MobileWorld Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. Everyday Dec 2025 55.3% In use
NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. Expert Dec 2025 68.2% In use
OctoCodingbench Agents complete 72 coding tasks while following rules from system prompts, AGENTS.md files, skills, and memory. An LLM judge checks 2,422 pass or fail items. University Dec 2025 36.2% Fading
Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. Expert Dec 2025 1.44 System cards only
Pieverse Agent Prediction Arena Agents trade event contracts on Polymarket with $1,000 each in real money. Pieverse tracks account value as market prices change. University Dec 2025 $981 Not in cards
PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. Expert Dec 2025 99.6% System cards only
REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. Everyday Dec 2025 53.1% Not in cards
SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. Everyday Dec 2025 57.8% Not in cards
SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. Expert Dec 2025 $548,400,000 Not in cards
SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. Expert Dec 2025 33.8% In use
Solubility Optimization (anindya) Models choose chemical transformations to improve predicted solubility. The package includes a benchmark runner that measures improvement on evaluation examples. University Dec 2025 Not in cards
SpatialBench (LatchBio) Agentic spatial-transcriptomics data analysis problems with deterministic graders. Anthropic reports a 115-problem Verified variant. Expert Dec 2025 57.6% System cards only
SpatialBench (original study) Agents complete 146 tasks in spatial transcriptomics. The authors compare their answers with results from the original analyses. Expert Dec 2025 38.4% Not in cards
SpatialTree Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level. Everyday Dec 2025 58.1% System cards only
SpreadsheetBench Verified Models solve 400 real spreadsheet manipulation questions from Excel forums, an expert annotated subset of SpreadsheetBench. Each answer runs against several test spreadsheets. University Dec 2025 79.1% Fading
SusVibes Coding agents rebuild a feature in a real open-source Python project from a plain-language request. Each task comes from a historical vulnerability fix, so the original human version was insecure. Hidden functional tests and security tests check the result. Scores report the share of solutions that work and the share that are also secure. Expert Dec 2025 43.5% Not in cards
SWE-EVO Coding agents implement release-note changes in seven mature Python projects, editing about 21 files per task. Large test suites check each of the 48 tasks. Expert Dec 2025 39.6% Fading
Tau2 Bench Verified Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. University Dec 2025 Not in cards
tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. Everyday Dec 2025 82.7% Fading
Terminal-Bench Pro Agents solve 400 terminal tasks across eight domains. Harbor distributes the 200 public tasks. Alibaba keeps the other 200 for private evaluation. Expert Dec 2025 51% Not in cards
VIBE (MiniMax) Agent builds complete web, simulation, Android, iOS or backend apps from scratch. Agent verifiers check execution, interaction and visuals. Expert Dec 2025 90.7% Fading
WebArena-Verified Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. Everyday Dec 2025 69% In use
WOLF Models play Werewolf to measure deception production and deception detection as separate abilities. Everyday Dec 2025 Not in cards
WorldTravel Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data. Everyday Dec 2025 32.7% Fading
A.S.E 2.0 Models and agentic coding tools write code in real projects covering 29 CWE types. A mix of static analysis, tests and vulnerability proofs of concept checks security. Expert Nov 2025 Not in cards
AndroidDaily Agents complete 350 daily tasks in 94 real closed-source Android apps. A process-aware reviewer model checks each trajectory against written guidelines. Everyday Nov 2025 61.9% Fading
ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. Expert Nov 2025 66.8% System cards only
Audio Reasoning Challenge 2026 Models answer questions about speech, music, and environmental audio and explain their reasoning. Incorrect answers score zero. A model judge grades correct answers for reasoning quality, with the middle three of five evaluation runs averaged. Everyday Nov 2025 Not in cards
Augmented Reality Patent Search Agents search 12,000 augmented and virtual reality patent applications to answer technical questions. A model judges each answer. University Nov 2025 Not in cards
Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. Expert Nov 2025 73.7% System cards only
BrowseSafe-Bench Detectors and browser agents face prompt injections hidden in realistic HTML pages, across 11 attack types, nine placements and three writing styles. 3,680 test samples. University Nov 2025 86.3% Not in cards
CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. Expert Nov 2025 System cards only
Code Arena Models act as coding agents that build web apps from user prompts, and users vote between two results. Votes yield Elo-style ratings. Nov 2025 1,826 In use
CodeClash Models build and improve their own codebases over many rounds, then compete in arenas such as Halite, poker and BattleSnake. Results give Elo ratings. Expert Nov 2025 1,385 Not in cards
Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. Expert Nov 2025 0% System cards only
DeepAmbigQA Search-augmented models answer multi-hop questions that combine entity-name ambiguity with complete answer-set retrieval. University Nov 2025 Not in cards
Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. Expert Nov 2025 90% System cards only
Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. Everyday Nov 2025 3.60 System cards only
Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. Expert Nov 2025 0% System cards only
Mathematical Problem Repository for AlphaEvolve Agents generate programs that construct mathematical objects or improve known bounds. The repository documents 67 problems and provides verification code for many of them. Research frontier Nov 2025 Not in cards
MCP Atlas Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. University Nov 2025 90.7% In use
Mini Amusement Parks Agents build rides, hire staff, and manage an amusement park. The simulator scores the park’s final value. Everyday Nov 2025 371375.0 Not in cards
Multi-agent search (Anthropic internal benchmark) Agents solve difficult information retrieval problems with web search, alone or as orchestrators that delegate to subagents. Expert Nov 2025 74.8% System cards only
NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. Expert Nov 2025 Not in cards
OASIS (agent safety) Agents handle multi-turn requests with different levels of complexity. Evaluators measure the proportion of safe responses. University Nov 2025 95.2% Not in cards
OSGym Infrastructure running 1,000+ parallel OS sandboxes for computer-use agent training and evaluation. Tasks span browsing, documents, SWE and multi-app work. Everyday Nov 2025 Not in cards
Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. Expert Nov 2025 0% System cards only
Prompt injection robustness against adaptive attackers, coding (Anthropic) Coding agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Expert Nov 2025 0% System cards only
Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Expert Nov 2025 0% System cards only
PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. Expert Nov 2025 10.5% System cards only
Real-Time Reasoning Gym Models act in games while token generation consumes part of the game clock. Everyday Nov 2025 0.81 Not in cards
ResearchRubrics 101 deep-research prompts with 2,593 expert-written weighted rubric criteria (including penalties). Reports graded for grounding, reasoning and clarity. Expert Nov 2025 81% In use
SWE-fficiency 498 tasks in nine Python data/ML/HPC repos: agent must speed up a real workload without breaking tests. Scored as speedup relative to expert patch. Expert Nov 2025 22.5% In use
Tau2 Bench Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. University Nov 2025 Not in cards
Terminal-Bench 2.0 89 hard terminal tasks across 16 categories in containers. Hidden tests verify the final state. 5 trials per task. Expert Nov 2025 82.3% In use
TIR-Bench Models solve 13 kinds of image tasks that need tool use on images, such as rotation, jigsaw, and maze. Most items are multiple choice. Everyday Nov 2025 61.6% Fading
Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. Everyday Nov 2025 $16,843 In use
Vending-Bench Arena Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance. Expert Nov 2025 $15,962 Not in cards
Vibe Code Bench v1 Agents build web applications from scratch from a natural-language spec in OpenHands. The built apps are tested against functional requirements. Expert Nov 2025 41.3% Not in cards
WebTailBench Computer-use agents complete 609 hand-verified tasks on live websites across 11 categories, such as bookings, real estate and job search. A companion set of 111 harmful tasks tests refusals. Everyday Nov 2025 57.4% Fading
WebTailBench-Refusals Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused. Everyday Nov 2025 100% System cards only
WISER-FindAll Entity discovery systems return all entities that match complex multi-criteria web queries. Expert Nov 2025 21% Not in cards
AgentBench FC Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format. Everyday Oct 2025 58.9% Not in cards
Alpha Arena Frontier models trade live markets with real money over a season, and the leaderboard ranks them by returns. Expert Oct 2025 Not in cards
Biomni-Eval1 Agents use biomedical data and tools to answer 433 questions across ten task types. Each task checks a structured answer against a reference. Expert Oct 2025 Not in cards
BuildArena Agents design machines in the Besiege simulator from written requirements. The simulator tests whether a vehicle reaches its target, how much weight a bridge supports, and how a lifting machine performs. University Oct 2025 45.3% Not in cards
BuilderBench Agents learn to control a simulated robot that builds target block structures through physical interaction and exploration. Expert Oct 2025 98% Not in cards
Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. Everyday Oct 2025 37% Not in cards
CATArena Code agents write programs that play board and card games in repeated tournaments. Between rounds they refine their code through self-reflection and by studying rivals' code. Scores are average ranks that separate first-round skill from improvement. Expert Oct 2025 1,799 Not in cards
CATArena Gomoku Code agents compete in Gomoku. The public competition board reports Elo Rating for each agent configuration. Expert Oct 2025 1,799 Not in cards
CATArena OIBench-5 Code agents compete in OIBench-5. The public competition board reports Time AUC for each agent configuration. Expert Oct 2025 0.40 Not in cards
CATArena Pommerman Code agents compete in Pommerman. The public competition board reports Elo Rating for each agent configuration. Expert Oct 2025 Not in cards
CATArena SWE-Perf-10 Code agents compete in SWE-Perf-10. The public competition board reports Time Ratio for each agent configuration. Expert Oct 2025 1.95× Not in cards
CATArena Tetris Code agents compete in Tetris. The public competition board reports Score for each agent configuration. Expert Oct 2025 Not in cards
CATArena Texas Holdem Code agents compete in Texas Holdem. The public competition board reports Elo Rating for each agent configuration. Expert Oct 2025 1,463 Not in cards
Cognition CodeSearch Eval Agents retrieve relevant files and line ranges for questions about real repositories. Cognition compares their answers with human labels using precision-weighted F scores. University Oct 2025 0.59 Not in cards
Cursor Bench (2025 internal version) Agents answer real agent requests from Cursor engineers in Cursor's harness. Graders compare against hand-curated solutions and check use of the codebase's existing abstractions. Expert Oct 2025 Fading
CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. Expert Oct 2025 Not in cards
Dojo Mini Bench Agents complete tasks in LinkedIn, Linear and Gmail through Dojo. Each task has its own completion check. Everyday Oct 2025 Not in cards
DPAI Arena Spring Benchmark Coding agents implement changes in Java Spring repositories. DPAI Arena scores each run and publishes its duration. Expert Oct 2025 64.9 Not in cards
FML-bench Agents propose and run experiments on eight machine learning research tasks. The authors measure task performance and how widely the proposals vary across iterations. Expert Oct 2025 0.50 Not in cards
Fruit Box Models select rectangles whose entries sum to ten on a ten-by-seventeen grid. The evaluator compares the final score with an expert trajectory from the training split. School Oct 2025 Not in cards
Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. Expert Oct 2025 41% System cards only
ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. Expert Oct 2025 0% System cards only
LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. Everyday Oct 2025 69.6% Not in cards
LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. Everyday Oct 2025 53% Not in cards
LiveResearchBench Deep research agents write cited reports for user-centric questions that need current information from many live web sources. Rubric-based judges score the reports. Expert Oct 2025 73.7 Not in cards
Magentic Marketplace Agents search for sellers, exchange proposals and make purchases. The simulation measures how much value customers receive after paying for their purchases. Everyday Oct 2025 Not in cards
Mastermind (Prime Intellect) Models guess a hidden sequence and receive feedback after each turn. The default evaluation uses 50 games and rewards success, speed, and deductions. Everyday Oct 2025 1.62 Not in cards
MEMTRACK Agents track changes across software platforms and use earlier information to complete tasks. Everyday Oct 2025 61% Not in cards
Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. Expert Oct 2025 8.30% System cards only
Next.js Agent Evals Coding agents with their models do real Next.js tasks. The board reports success rate with and without an AGENTS.md file, plus duration and cost. Expert Oct 2025 100% Not in cards
OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. University Oct 2025 61.8% Fading
OpenGameEval Agents write Luau scripts and edit objects in Roblox Studio. The original suite checks 47 changes to game mechanics and scenes. University Oct 2025 55.3% Not in cards
OSWorld-MCP Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps. Everyday Oct 2025 56.6% Not in cards
OutboundEval Agents conduct outbound calls with simulated users. The benchmark scores compliance with the professional workflow and the quality of the interaction. Expert Oct 2025 Not in cards
Petri Open-source auditing tool where auditor agents probe target models in simulated scenarios and judges score misaligned behaviors. Expert Oct 2025 System cards only
PRDBench Code agents build 50 real-world Python projects across 20 domains from product requirement documents. A trained judge agent checks each project against structured criteria. Expert Oct 2025 69% Not in cards
Prophet Arena Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return. Expert Oct 2025 86.2 In use
RECODE-H Agents implement research code from papers and repositories. They can request feedback from a simulated collaborator before their code runs against tests. Expert Oct 2025 84.3% Not in cards
Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. Expert Oct 2025 41.3% In use
ReplicationBench Agents reproduce computational results from published astrophysics papers. Evaluators check their numerical answers and assess whether their methods follow the original work. Research frontier Oct 2025 22% Not in cards
RoboChallenge Fine-tuned policies run 10 real rollouts on each of 30 tabletop tasks across four robot types. Scored by success rate and progress. Everyday Oct 2025 42.7% Not in cards
Sherlock Holmes Q&A Models search 55 Sherlock Holmes stories and answer 140 literary comprehension questions. A model judge checks their answers. School Oct 2025 100.0 Not in cards
SQL-API-Bench Agents answer questions that require both database queries and API calls. The evaluator checks answers across two mixed database and API benchmark settings. Expert Oct 2025 Not in cards
StockBench Agents trade stocks over a fixed evaluation window. StockBench ranks models by the mean of final return, maximum drawdown and Sortino ratio. The score is the board's rank, so a lower score is better. Expert Oct 2025 1.00 Not in cards
Toolathlon 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. University Oct 2025 56.5% In use
USIM underwater robot tasks Agents follow instructions to navigate, grasp, transport, and track objects underwater. The simulator measures task completion across 700 trials. Everyday Oct 2025 37.6% Not in cards
Vals Index v1 Vals AI's first index of agentic work in finance, law and coding, with each sector weighted by its share of U.S. GDP. Oct 2025 72.2% System cards only
Vals Multimodal Index v1 Models complete finance, coding, and education tasks that contribute to a weighted index. Expert Oct 2025 71.1% Not in cards
VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. University Oct 2025 44.8% Not in cards
WARC-Bench Screenshot-only agents do short web subtasks, such as date pickers and scroll containers, on real sites replayed from Web Archive snapshots. A program checks the end state. Everyday Oct 2025 64.8% Not in cards
World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. Everyday Oct 2025 64.8% Not in cards
xbench-DeepSearch-2510 Agents answer 100 new Chinese deep search questions that need many web searches and cross-checks. Answers are matched to a key. Expert Oct 2025 56.3% Fading
AI Spreadsheet Benchmark Spreadsheet assistants complete 53 prompts for analysis, charts, formulas, and workbook management. Checks measure task success and whether outputs update when source data changes. The evaluation reports pass@1, pass@3, dynamic output rate, and latency. University Sep 2025 Not in cards
Artificial Analysis Intelligence Index v3 Version 3 of Artificial Analysis's index, ten evaluations that added agentic terminal and tool-use tasks to the knowledge, math and coding tests. Sep 2025 61.0 Dropped
Automotive-ENV Agents operate climate controls in an Android Automotive OS simulator. A validator checks the screen and vehicle signals after requests such as turning on the defroster. Everyday Sep 2025 49.3% Not in cards
BEHAVIOR Challenge 2025 50 long-horizon household activities on a Galaxea R1 Pro in OmniGibson, trained from 10,000 demos. Scored by Q-score and success. Everyday Sep 2025 Fading
BioML-bench Agents build and evaluate machine learning systems for biomedical experiments spanning proteins, drugs, single cells, imaging, and biomarkers. Task-specific metrics compare submissions with experimental reference results. Expert Sep 2025 Not in cards
Blackjack Basic Strategy Models play blackjack hands or select a single move from a dealt position. The grader measures the expected value of each decision against basic strategy. Everyday Sep 2025 Not in cards
BlueBench Agents solve enterprise tasks across knowledge, coding, security and business workflows. The benchmark aggregates capability scores from multiple public task suites. Expert Sep 2025 64.6 Not in cards
Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. Expert Sep 2025 51.3% Dropped
CTF-Archive Agents solve capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. Expert Sep 2025 Not in cards
DeepResearch Arena Deep research agents get research tasks generated from academic seminar transcripts, which reflect expert discussion and reduce data leakage. Expert Sep 2025 3.96 Not in cards
DRBench Agents investigate open-ended business questions across enterprise applications and produce reports. The evaluator checks whether the reports recover relevant information and support their claims. University Sep 2025 63.8 Not in cards
Earth-Bench Agents analyze satellite observations with Earth science tools. The evaluator checks the final answer and compares tool calls with a reference sequence. University Sep 2025 66% Not in cards
FinSearchComp Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. Expert Sep 2025 81.7% In use
FinSearchComp Global Agents search for financial information about global assets across three task types. The score is the published average accuracy for the Global subset. Expert Sep 2025 65.5% Dropped
FinSearchComp Greater China Agents search for financial information about Greater China assets across three task types. The score is the published average accuracy for the Greater China subset. Expert Sep 2025 Not in cards
ICPC World Finals 2025 Models solve the 12 problems of the 2025 ICPC World Finals under contest time and attempt limits. Scored by problems solved. Competition Sep 2025 10.0 Fading
IMProofBench Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically. Research frontier Sep 2025 49.3% Not in cards
Lights Out 3×3 Models toggle lights on a three-by-three grid until every light is off. The dataset reserves 30 boards for testing and rewards efficient solutions. School Sep 2025 Not in cards
LiveNewsBench Models search for answers to questions about recent news. The test limits searches and page visits and compares answers with curated references. Everyday Sep 2025 21.5% Not in cards
Malicious use of Claude Code (Anthropic) Agents in Claude Code receive malicious and dual-use coding requests. Score is the refusal rate on malicious requests. Expert Sep 2025 96.7% System cards only
MAPP (Multi-Agent Path Planning) Models move several agents across a grid without collisions. The environment tracks how many agents reach their goals and how closely their routes follow shortest paths. School Sep 2025 70.8% Not in cards
Mastermind Information Gain Models choose a Mastermind move from earlier guesses and feedback. The evaluator measures information gain for individual moves and success over complete games. School Sep 2025 Not in cards
MCP prompt injection evaluation (Anthropic) Agents use simulated email, Slack and document servers over MCP where content carries injected instructions. Score is the attack prevention rate. Expert Sep 2025 92.5% System cards only
MCPMark Agent stress-tests realistic MCP tool use (GitHub, Notion, Postgres, browser). Graded by programmatic verification scripts. University Sep 2025 94.5% In use
Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. Everyday Sep 2025 42.1% Fading
Multi-Turn Sudoku (m8ngotree) Models fill a nine-by-nine Sudoku board one move at a time. The generator checks for a unique solution, and the scorer rewards valid moves and completion. School Sep 2025 Not in cards
OceanGym Agents interpret underwater images and sonar, then navigate simulated search and inspection tasks. The simulator scores how closely they reach target locations. University Sep 2025 14.6 Not in cards
OOLONG Models aggregate information across long collections of text. The hub port implements the synthetic and real datasets with a context-controlled recursive language model interface. University Sep 2025 68.3 Not in cards
Project Euler (MathArena) Models solve recent Project Euler problems that need both mathematical insight and code. MathArena adds problems as Project Euler releases them, starting from problem 943. Expert Sep 2025 89% Not in cards
Red Team Defense Simulator Models respond to an adaptive adversary. The evaluation measures successful defense, helpfulness and false refusals. University Sep 2025 85% Not in cards
Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. Expert Sep 2025 System cards only
robust-kbench Agents write CUDA kernels that are tested across several weight initializations and input shapes, so shape and magnitude exploits fail. Correctness checks and profiling score each kernel. Expert Sep 2025 Not in cards
SCUBA Agents work in a Salesforce sandbox as administrators, sales representatives, or service agents. Evaluators check milestones such as creating accounts and routing support cases. University Sep 2025 53.9% Not in cards
Security Configuration Verification Agents audit Kubernetes and Terraform configurations, identify security violations and submit fixes. The grader weights detected violations by severity and validates patches. University Sep 2025 Not in cards
SWE-Bench Pro 1,865 long-horizon issue-resolution tasks from 41 repos in four languages. Agent patch must pass hidden tests. Pass@1. Expert Sep 2025 61.5% In use
SWE-Bench Pro Private Agents resolve held-out software issues from the private SWE-Bench Pro split. The leaderboard reports the percentage of resolved tasks. Expert Sep 2025 51.5% Not in cards
Terminal-Bench Hard Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. Expert Sep 2025 36% In use
Tool use prompt injection evaluation (Anthropic) Agents run bash commands where files, scripts or command outputs carry injected instructions. Score is the attack prevention rate. Expert Sep 2025 96% System cards only
UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. Everyday Sep 2025 59.8 Dropped
Valyrian Games Coding Challenge Models create coding problems with integer answers and repeatedly solve their own problems to qualify. Tournament entrants solve one another’s problems and receive TrueSkill ratings. Competition Sep 2025 30 Not in cards
VisualProbe Models search high-resolution images for small targets to answer questions, often over many crop and zoom turns. Answers are checked against a key. Everyday Sep 2025 65% Fading
VITA-Bench Agents handle multi-turn requests in simulated food delivery, in-store, and travel services with 66 tools. A rubric-based judge scores task success. Everyday Sep 2025 32.5% Fading
VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. Everyday Sep 2025 72.1% Not in cards
X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. Expert Sep 2025 58% Dropped
AI FPL Models manage Fantasy Premier League squads using the same weekly prompt. Their transfers, captain choices, and starting lineups earn points from real matches. Everyday Aug 2025 Not in cards
Android Agent Arena (100 tasks) Agents use 20 Android apps to find information and perform actions. A vision model checks whether the agent reached the required intermediate states and completed the task. Everyday Aug 2025 53% Not in cards
AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. Expert Aug 2025 58% Not in cards
BalatroBench Models play the deck-building roguelike Balatro through an API bot. The leaderboard compares how far each model and strategy gets in a run. Everyday Aug 2025 19.3 Not in cards
Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. Everyday Aug 2025 0.33% In use
Brokk Power Ranking Models solve 181 real tickets from a 1.1 million line Rust codebase. Model judges compare solutions in pairs, and a Bradley-Terry fit gives each model a strength score. Expert Aug 2025 35.4 Not in cards
BrowseComp-Plus Deep research agents answer 830 BrowseComp questions by searching a fixed corpus of about 100,000 human-verified documents. Expert Aug 2025 70.1% In use
BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. Expert Aug 2025 59% Fading
Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. Expert Aug 2025 99% System cards only
Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. Expert Aug 2025 1.30% System cards only
CompileBench Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code. Expert Aug 2025 100% Not in cards
ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. Everyday Aug 2025 66.5% Dropped
CUA-SHADE-Arena Monitors review computer-use agent activity in simulated workplace tasks. The evaluation measures how often they flag unsafe behavior at a fixed false positive rate. University Aug 2025 Not in cards
DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. Everyday Aug 2025 39% Not in cards
FutureX Agents predict upcoming events collected daily from 195 websites in 11 domains, within a one-week window. Answers are scored once real results are crawled, with harder tiers weighted more. Expert Aug 2025 27.9 Not in cards
Game Arena Chess Language models play full chess games against each other with the board given as FEN and PGN text. Ratings come from many games between each pair. Expert Aug 2025 1,646 Not in cards
GEN-Bench Agents select job operations as conditions change in a simulated factory. The benchmark compares schedule completion times with the best known result for each instance. University Aug 2025 6.09% Not in cards
GitTaskBench Code agents solve 54 real tasks across 7 modalities and 7 domains by using a given GitHub repository, from understanding and setup to running it. Expert Aug 2025 48.1% Not in cards
Hallucinations on production-representative prompts Models answer prompts representative of ChatGPT production traffic. A grader with web access scores incorrect claims and responses with major errors. Everyday Aug 2025 3.10% System cards only
HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. Expert Aug 2025 16% Fading
IOI (Vals) Agents solve International Olympiad in Informatics problems from 2024 and 2025 in C++. A submission tool modeled on the contest grades up to 50 tries per problem by subtask. Competition Aug 2025 91.7% Not in cards
Kaggle Game Arena Language models play each other in board, card and word games through a text harness. One leaderboard ranks models across all Game Arena games. Expert Aug 2025 638 Not in cards
LiveDRBench Agents perform 100 deep research tasks and return key claims discovered during search. Scores use claim-level F1. Expert Aug 2025 55.5% Not in cards
LiveTradeBench Agents allocate simulated portfolios across US securities and Polymarket contracts using streamed prices and news. The researchers track portfolio returns over successive trading days. University Aug 2025 12.1% Not in cards
LLM Chess Arena (Louis Guichard) Models play each other from a text diagram, piece locations, and recent moves. A model forfeits after three failed attempts to make a legal move. School Aug 2025 1,495 Not in cards
M3-Bench Multimodal agents answer questions about long videos while using long-term memory and multimodal retrieval. Expert Aug 2025 Not in cards
MCP-Bench Agents solve multi-step tasks using 28 live MCP servers with 250 tools in areas such as finance, travel, scientific computing and academic search. Scoring covers tool use, coordination and planning. Expert Aug 2025 0.75 Not in cards
MCP-Universe Agents complete tasks through 11 real MCP servers across six domains, such as navigation, repositories, finance, and browsing. Scored by execution-based success rate. University Aug 2025 44.6% Fading
MCPTox Agents use tools whose descriptions contain untrusted instructions. The benchmark measures whether agents follow those instructions across 1,348 cases from 45 servers. University Aug 2025 5.10% Not in cards
MLE-bench 30 Agents train models for 30 of MLE-bench's Kaggle-style competitions. The score is the share of runs that earn a bronze medal or better. Expert Aug 2025 36.7% System cards only
MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. Expert Aug 2025 24.3% Fading
MMSearch-Plus Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key. Expert Aug 2025 41.4% Fading
OdysseyBench Agents recover requirements from conversation histories and complete workflows across documents, spreadsheets, PDFs, email, and calendars. Task-specific checks inspect the resulting file system and report the percentage of completed tasks. University Aug 2025 58.7% Not in cards
OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. Research frontier Aug 2025 8.00% System cards only
Recovery-Bench Agents repair a task after a weaker agent leaves failed actions and a corrupted environment. The score is the share of recovery tasks that pass the original task verifier. Expert Aug 2025 Not in cards
ReportBench Agents write literature surveys. The evaluator compares their references with published surveys and checks whether cited sources support their claims. Expert Aug 2025 0.39 Not in cards
Research-Eval Search-augmented models answer 374 grounded multi-hop questions that need sources across many kinds of websites. An LLM judge checks each answer against a checklist of requirements. Expert Aug 2025 66% Dropped
ScrabbleBench Models choose a word from a Scrabble rack and board. A solver finds its best legal placement. The score averages the four seats in self-play. Everyday Aug 2025 152.8 Not in cards
Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. Expert Aug 2025 99% System cards only
TRAJECT-Bench Agents select and execute tools to answer queries. The evaluation checks whether the actions satisfy a reference trajectory and produce the correct answer. University Aug 2025 70.5% Not in cards
UQ Models answer 500 unsolved Stack Exchange questions. A model checks answers without a reference solution, and community members review selected answers. Research frontier Aug 2025 Not in cards
WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Everyday Aug 2025 83.9% In use
Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. Expert Jul 2025 67% System cards only
AlgoTune Agents rewrite code for 154 math, physics, and computer science functions to run faster and stay correct. Scoring is harmonic mean speedup over reference code. Expert Jul 2025 2.05× Not in cards
Asper-Bench Agents turn complex user requests into programs that execute multi-step actions in a simulated digital assistant environment. Automated checks compare the resulting state with the requested goal. Expert Jul 2025 Not in cards
BFCL V4 Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Everyday Jul 2025 75% In use
CC-Bench Claude Code runs realistic frontend, backend, tooling and data coding tasks in fresh containers. Humans compare models head-to-head (win rate). Expert Jul 2025 41.9% System cards only
Emergent Price-Fixing by LLM Auction Agents Models trade in simulated auctions and communicate with other participants. A judge examines game logs for evidence of price fixing. Everyday Jul 2025 19% Not in cards
FantasyFootballBench Models draft players, set weekly lineups, and negotiate trades in a replay of the 2024 NFL season. Actual player statistics determine points and wins. Everyday Jul 2025 15.0 Not in cards
Galileo Agent Leaderboard v2 Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality. Everyday Jul 2025 62% Not in cards
Gray Swan Agent Red Teaming (ART) benchmark Agents face curated prompt injections from a large public red-teaming competition across 44 deployment scenarios. Score is attack success rate. Expert Jul 2025 0.10% System cards only
Group Binary Search (Riedl) Agents each choose an integer without seeing the other guesses. They adjust after hearing whether the group sum exceeds or falls below a hidden target. Everyday Jul 2025 75% Not in cards
Husky Hold'em Bench Models write Python poker bots and revise them after practice games. Six bots compete at each table, and the tournament tracks average winnings. University Jul 2025 Not in cards
In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. Expert Jul 2025 78% System cards only
Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. Expert Jul 2025 88.6% Fading
Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. Expert Jul 2025 99.5% System cards only
Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. Expert Jul 2025 95% System cards only
Labelbox Deep Research Agents Agents investigate research questions at the doctoral level. Labelbox compares their answers and reports scores by domain. Expert Jul 2025 77.5% Not in cards
LLM Economist Models set tax rates or choose working hours in a simulated economy. Researchers compare the social welfare achieved under different tax policies. University Jul 2025 Not in cards
MMBench-GUI GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration. Everyday Jul 2025 81.5% System cards only
Multi-SWE-bench flash Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing. Expert Jul 2025 25.3% Dropped
OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. Everyday Jul 2025 53.4% Not in cards
OSWorld-Verified 369 open-ended tasks in real desktop apps. Agent acts through screenshots, mouse and keyboard. Execution-based checkers score success. Everyday Jul 2025 86% In use
PACT Two models bargain over 20 rounds as buyer and seller, each with a hidden private value. Matches across many pairings rank the models. Everyday Jul 2025 1,607 Not in cards
RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. Everyday Jul 2025 44.3% Fading
SLDBench Agents infer scaling laws from data covering more than 5,000 language model experiments. Eight tasks test how well their formulas predict results at other scales. Expert Jul 2025 Not in cards
SnorkelWordle Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate. Everyday Jul 2025 94% Not in cards
SWE-Perf Agents speed up code in 140 tasks drawn from performance pull requests in real GitHub repos. Scored on correctness and runtime gain. Expert Jul 2025 2.26% Fading
TextQuests Agents play 25 classic interactive fiction games. The score is harmful actions per 100 steps, where lower is better. The board also reports game progress. Everyday Jul 2025 7.00 System cards only
UserBench Agents help simulated users who start with vague goals and reveal preferences bit by bit. Agents must ask clarifying questions and use tools to reach choices that fit the user. Everyday Jul 2025 0.61 Not in cards
WISER-Search Web research agents answer fresh and business questions that require current sources, multi-hop search, and evidence synthesis. Expert Jul 2025 Not in cards
Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. Expert Jun 2025 0% In use
AIRTBench Models act as autonomous AI red teamers on 70 black-box capture-the-flag challenges from Dreadnode's Crucible platform. Each challenge hides a flag behind an AI or ML system, and the model writes Python code to attack it with methods such as prompt injection, evasion and model inversion. A challenge counts as solved when the model submits the correct flag. Expert Jun 2025 46.9% Not in cards
ALE-Bench Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants. Competition Jun 2025 3,164 Not in cards
AssetOpsBench Agents automate industrial asset work, such as condition monitoring, maintenance planning and intervention scheduling, using sensor data, failure models and work orders. Expert Jun 2025 65% Not in cards
Automated LLM Speedrunning Benchmark Agents try to reproduce 19 past records of the NanoGPT speedrun, given the earlier code and hints of varying detail. Scored by how much of each record's speedup they recover. Research frontier Jun 2025 46% Not in cards
BashBench Agents complete multi-step system administration tasks while red teams attempt covert malicious actions in the same shell environment. Expert Jun 2025 Not in cards
Bench to the Future Agents forecast hundreds of already-resolved questions while researching an offline corpus of web pages saved before the resolution. Scored by Brier score. Expert Jun 2025 Not in cards
BIRD-Interact Models solve ambiguous database tasks by talking with a simulated user, in a conversational mode and an agentic mode. 600 BI and CRUD tasks are checked by executable tests. Expert Jun 2025 25.5% Not in cards
CVDP Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs. Expert Jun 2025 64.1% Not in cards
CyberGym Large-scale benchmark of real open-source vulnerabilities. Agent must reproduce each known vulnerability with a proof-of-concept input. Expert Jun 2025 79.7% In use
DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. Expert Jun 2025 Not in cards
DeepResearch Bench Agents write cited research reports for 100 PhD-level tasks across 22 fields. Judges score report quality against references and check citation accuracy. Expert Jun 2025 55.3% Fading
DeepShop Web agents answer complex shopping queries on live sites with product attributes, search filters and sorting preferences. An LLM judge checks whether results meet each requirement. Everyday Jun 2025 42.3% Fading
FieldWorkArena V2.0 Agents inspect workplace images, videos, manuals, and logs to plan actions and report operational events. The evaluator compares their answers and actions with reference results. It combines correctness judgments with partial credit for numerical agreement. University Jun 2025 52% Not in cards
Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. Expert Jun 2025 53.1% Not in cards
GeoBenchX v2 Agents use GIS tools to join data, measure spatial relationships, and draw maps. Three model judges compare their work with corrected reference solutions. University Jun 2025 84% Not in cards
Hangman Bench Agents guess letters to identify hidden words in Hangman. The harness tracks wins across difficulty levels. Everyday Jun 2025 Not in cards
HexMachina Models write Catan players in code and compete in the board game. University Jun 2025 Not in cards
Labelbox Agentic Search Agents search for answers to 200 questions across several knowledge domains. Labelbox reports overall and domain scores. University Jun 2025 80% Not in cards
MCP-SIM Agents turn physics problems into FEniCS simulations. The tasks include heat flow, elastic deformation, and fracture. The authors count solved tasks. University Jun 2025 Not in cards
Mind2Web 2 Agents browse the live web for 130 long tasks and return answers with citations. Task-specific judge agents built from rubric trees grade correctness and attribution. Expert Jun 2025 28% Not in cards
OPT-BENCH Agents iteratively improve solutions to 20 machine-learning tasks and 10 combinatorial optimization problems using execution feedback. Win counts, invalid-solution rates, ranks, and improvement rates measure optimization performance. University Jun 2025 0.69 Not in cards
Orak LLM agents play 12 popular video games across most genres through an MCP interface. Each game scores in-game progress or outcomes, and the board ranks models by their average rank across the games. Everyday Jun 2025 3.90 Fading
RExBench Agents extend 12 published AI research projects from expert instructions. The evaluator runs their code against task-specific success criteria. Harbor distributes two public tasks. Research frontier Jun 2025 50% Not in cards
RoboArena Double-blind pairwise A/B trials of generalist DROID policies on real Franka arms, free-form tasks by volunteer evaluators. Elo-style rating. Everyday Jun 2025 1,608 Not in cards
RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. Everyday Jun 2025 58.3% Not in cards
ScreenSuite Models use screenshots to locate controls, select actions and complete computer tasks. The suite measures perception, single actions and complete interactions. Everyday Jun 2025 56% Not in cards
Seal-0 Models answer 111 fact-seeking questions where web search returns conflicting or unhelpful results. Answers are graded for accuracy, usually with search enabled. University Jun 2025 57.4% Fading
SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. Expert Jun 2025 18% Not in cards
SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. Jun 2025 26.2% System cards only
tau2-bench Dual-control customer-service simulations (retail, airline, telecom) where agent and simulated user both act. Graded on final state. Everyday Jun 2025 98.9% In use
ts-bench Coding agents such as Claude Code, Codex and Gemini CLI solve TypeScript tasks, and tests decide pass or fail. Version 1 uses 25 Exercism practice exercises. Version 2 uses real tasks from a large TypeScript monorepo, taken from SWE-Lancer. University Jun 2025 80% Not in cards
ts-bench v1 Coding agents solve 25 self-contained TypeScript exercises from Exercism. The score is the percentage of exercises that pass their tests. University Jun 2025 96% Not in cards
V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. Everyday Jun 2025 Not in cards
Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. Everyday Jun 2025 84.9% Not in cards
WorldPrediction Agents predict and plan in procedural environments with long horizons. Tasks test whether models maintain a useful world model while acting. University Jun 2025 Not in cards
xbench-Marketing Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers. Expert Jun 2025 50.8 Not in cards
xbench-Recruitment Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5. Expert Jun 2025 78.5 Not in cards
ADE-bench Agents repair dbt projects so that SQL tests pass on the resulting tables. Harbor ports 48 DuckDB tasks and excludes the Snowflake and dbt Fusion variants. Expert May 2025 Not in cards
Agentic coding malicious requests (Anthropic) Coding agents with standard coding tools receive 150 malicious coding requests prohibited by policy. Score is the refusal rate. Expert May 2025 100% System cards only
AndroidWorld-Knowledge Agents attempt 48 AndroidWorld tasks across five apps. Each app provides guidance for its simplest task, and harder tasks test whether the agent can reuse it. Everyday May 2025 Not in cards
BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. Expert May 2025 50% Not in cards
C3-Bench Agents complete multi-task tool use dialogues built to stress tool dependencies, long context dependencies and frequent switches between policy types. University May 2025 12.5% Dropped
Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. Expert May 2025 100% System cards only
Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy. Expert May 2025 Dropped
CRMArena-Pro Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private. Everyday May 2025 56.2% Not in cards
Deep Research Bench (FutureSearch) Web research agents work on multi-step research tasks against RetroSearch, a frozen copy of 10,000 to 100,000 web pages per task. Answers are compared with answers worked out by people. Expert May 2025 0.55 Not in cards
DeepConsult Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer. Expert May 2025 61.1% Fading
DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. Expert May 2025 80% System cards only
End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score. Expert May 2025 Dropped
EXP-Bench Agents get a research question and incomplete starter code from an influential AI paper. They must form hypotheses, design and run experiments, and analyze results. Research frontier May 2025 0.50% Not in cards
FoamBench Agents create OpenFOAM cases from written specifications. The grader checks whether the solver finishes and whether the final flow field matches a reference simulation. University May 2025 22.5% Not in cards
GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. Expert May 2025 90.9% System cards only
GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. Expert May 2025 100% System cards only
gg-bench Models play 126 games invented by a language model and receive game-specific outcomes. Everyday May 2025 36.3% Not in cards
GSO-Bench Agents speed up code in 102 tasks across 10 real codebases, given a performance test. Scored by the share matching the expert commit's speedup. Expert May 2025 88.2% Not in cards
Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. Expert May 2025 30% System cards only
Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. Expert May 2025 0.65 System cards only
Kernels task (Anthropic) Agents optimize compute kernels in a performance engineering challenge. Scored by speedup. Expert May 2025 System cards only
KORGym Models play 51 text and visual games designed to test interactive reasoning. Everyday May 2025 0.82 Not in cards
LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. Everyday May 2025 72.4% Fading
Letta Leaderboard Agents read, write and update core and archival memory while answering synthetic factual questions. A model judge scores answers and penalizes unnecessary memory operations. Expert May 2025 93.0 Not in cards
lmgame-Bench Models play platformer, puzzle and narrative games such as Sokoban, Tetris, 2048 and Candy Crush through a Gym-style API with perception and memory scaffolds. Everyday May 2025 87.7 Not in cards
Novel compiler task (Anthropic) Agents build a compiler for an unusual programming language from a specification and test cases. Expert May 2025 80.9% System cards only
ResearchCodeBench Agents read machine learning papers and fill missing Python code in the accompanying implementations. Unit tests check each of 212 snippets from 20 papers. Expert May 2025 64.2% Not in cards
Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. Expert May 2025 0% System cards only
RiOSWorld Computer-use agents work in an Ubuntu virtual machine through risky scenarios such as phishing emails and pop-ups. Scores track whether agents notice and avoid the risk. Everyday May 2025 79% Not in cards
ScienceBoard Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking. Expert May 2025 68% Not in cards
Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. Expert May 2025 System cards only
SnorkelUnderwrite Agents help simulated small business insurance underwriters across six task types in multi-turn chats, using tools and asking the underwriter for missing facts. Expert May 2025 91% Not in cards
Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). May 2025 93.3% System cards only
SWE-bench-Live Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch. Expert May 2025 71% Dropped
SWE-bench-Live Verified Task success on the official Verified split. The source reports each submission denominator separately. Other language and platform splits remain separate. Expert May 2025 40% Not in cards
SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail. Expert May 2025 66.6% Dropped
SWE-rebench Agents fix fresh GitHub issues that an automated pipeline mines each month. Tests decide whether the issue is resolved. Expert May 2025 64.5% System cards only
Terminal-Bench Agents complete about 80 hard tasks in sandboxed Linux terminals, such as compiling code or configuring servers. Test scripts check the final container state. Expert May 2025 49.6% Fading
Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. Expert May 2025 0.97 System cards only
ThinkGeo Agents use remote sensing tools to interpret aerial and satellite images. The authors compare individual tool calls and use GPT-4o mini to grade final answers. University May 2025 100% Not in cards
Time series forecasting task (Anthropic) Agents solve a regression and time-series forecasting problem with known state-of-the-art results, in six variants. Expert May 2025 4.55 System cards only
ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. Everyday May 2025 85.5% System cards only
Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. Expert May 2025 0.63% System cards only
Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. Expert May 2025 85% System cards only
VideoGameBench Vision-language models play 10 popular 1990s Game Boy and MS-DOS games in real time from raw frames. Scored by the share of each game completed. Expert May 2025 0.48% Not in cards
VideoGameBench Lite Agents play video games while the environment pauses during inference. The headline is the official Overall Score. Expert May 2025 1.60% Not in cards
WebBench (Halluminate) Browser agents complete 2,454 tasks on 452 live websites from the global top 1,000 by traffic. Tasks either read information or write data: creating, updating and deleting records, logging in and downloading files. The board compares browser agent products. Everyday May 2025 76.5% Not in cards
xbench-DeepSearch Agents search the web to answer 100 Chinese questions that need multi-step retrieval. Scored by accuracy against reference answers. Expert May 2025 83.7% Fading
AI Engineering Lab simulation self-validation Agents predict how a mechanical system should behave, inspect simulation outputs, and decide whether the model is correct. The authors score these decisions with F1. University Apr 2025 0.72 Not in cards
Among Us: A Sandbox for Measuring and Detecting Agentic Deception Language agents play a multiplayer social-deception game with hidden roles. Deception Elo and detection Elo measure success over repeated matches. Expert Apr 2025 Not in cards
Apollo Research sabotage evaluations Agents are told to sabotage another model through backdoors, sandbagging finetunes or hidden communication. Scored as mean task score from 0 to 1. Expert Apr 2025 0.17 System cards only
AutoCFD Incompressible Flow Benchmark Agents configure OpenFOAM simulations from flow descriptions and meshes. The authors check solver convergence and compare simulated flow with reference values. University Apr 2025 31.4% Not in cards
AutoPatchBench Agents write security patches for crashes that a fuzzer found in native code. Fixes are checked by fuzzing and differential debugging. Expert Apr 2025 11.5% Not in cards
BrowseComp Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. Expert Apr 2025 93.3% In use
BrowseComp-ZH Agents browse the Chinese web to answer 289 multi-hop questions with short, verifiable answers. Scored by accuracy. Expert Apr 2025 82.4% Fading
CRUST-bench Agents translate 100 C projects into safe Rust. Tests check whether each translation matches the supplied Rust interface and preserves the original behavior. Expert Apr 2025 48% Not in cards
Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. Expert Apr 2025 93.3% System cards only
DrafterBench Agents call drawing revision tools from written instructions. The evaluator records their operations and arguments across 1,920 tasks covering 12 types of revision. University Apr 2025 84.5% Not in cards
KUMO Agents make queries and use the responses to identify hidden facts in generated reasoning games. Harbor offers 5,050 easy games and 250 hard games. University Apr 2025 86.3% Not in cards
LLM-SRBench Models infer scientific equations from observations on 239 problems across four scientific domains. Symbolic accuracy checks whether the discovered equation recovers the underlying relationship. Research frontier Apr 2025 15.1% Not in cards
LMArena Search Arena People compare answers from two anonymous search-augmented models on their own questions and vote for the better one. Votes become Bradley-Terry ratings. Everyday Apr 2025 1,255 Fading
m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. Everyday Apr 2025 46.5% Dropped
MafiaBench Models play a social deduction game with hidden roles and public discussion. Everyday Apr 2025 1,561 Not in cards
MLRC-Bench Agents propose and implement methods for machine learning competitions. The scorer compares their performance with a supplied baseline and the strongest human solution. Research frontier Apr 2025 6.26 Not in cards
Multi-SWE-bench Agents resolve 1,632 GitHub issues in Java, TypeScript, JavaScript, Go, Rust, C, and C++ repos. Patches must pass the repos' tests. Expert Apr 2025 52.7% Fading
Multi-SWE-bench mini Agents fix real GitHub issues in 8 programming languages, a 400-instance subset of Multi-SWE-bench made for cheaper runs. Scored by tests passing. Expert Apr 2025 25.8% Dropped
Online-Mind2Web Web agents do 300 tasks on 136 live websites. An LLM judge reads the agent's trajectory and decides whether the task succeeded. Everyday Apr 2025 69% Fading
Open RAG Benchmark RAG systems answer 3,045 queries over 1,000 arXiv PDFs with text, tables, images, and hard negatives. Expert Apr 2025 Not in cards
PaperBench Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. Research frontier Apr 2025 93% In use
PaperBench Code-Dev Agents implement research paper code. The Code-Dev variant evaluates coding separately from full research replication. Research frontier Apr 2025 43.4% Not in cards
Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. Expert Apr 2025 System cards only
REAL Agents complete 112 tasks on replicas of 11 websites. State checks grade actions, while a model judge checks retrieved answers against references. Everyday Apr 2025 41.1% Not in cards
RepliBench Agents acquire resources, exfiltrate model weights, replicate onto compute and persist over time. Task checks measure components of autonomous self-replication. Research frontier Apr 2025 Not in cards
SWE-bench Multilingual 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Expert Apr 2025 96.6% In use
SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution. Expert Apr 2025 14.1% Not in cards
SWE-smith Agents repair software faults generated from real repositories. Harbor evaluates a 100-task sample from the larger training collection using tests for each fault. Expert Apr 2025 38% Not in cards
TextArena Collection of competitive text games for LLM eval and RL, with an online Elo leaderboard. Apr 2025 Not in cards
Vals Finance Agent v1 Agents answer 537 financial analyst questions on public company SEC filings using search and filing tools. An LLM judge grades each final answer. Expert Apr 2025 60.6% Dropped
WISER-Atomic Web research systems answer 121 customer-inspired questions across finance, sales, recruitment, and other domains. Expert Apr 2025 69% Not in cards
WixQA Models retrieve Wix support articles and answer customer questions from those articles. The authors score retrieval and answer generation separately. Everyday Apr 2025 Not in cards
AI Agent Privacy Web agents complete tasks while exposed to private information. The benchmark measures privacy leakage and unsafe disclosure during autonomous operation. Expert Mar 2025 Not in cards
AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. Expert Mar 2025 22% Not in cards
AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. Everyday Mar 2025 40% Not in cards
BLUR Models identify a remembered item from incomplete clues in several languages and media. Everyday Mar 2025 54% Not in cards
CVE-Bench Real-world web-application CVEs in sandboxed services. Agent is scored on reaching predefined attack objectives. Expert Mar 2025 94.1% System cards only
Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. Expert Mar 2025 91.7% System cards only
Dubesor AI Chess Leaderboard Models play chess by continuing game notation or by reasoning from the board and legal moves. The organizer maintains separate Elo ratings for each mode. School Mar 2025 1,907 Not in cards
Factorio Learning Environment Agents write Python programs that control a Factorio factory, in lab tasks with fixed resources and in open play. Scored by production throughput, milestones and lab task success. Expert Mar 2025 293206.0 Not in cards
Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. Everyday Mar 2025 Not in cards
GeoBenchX Agents use GIS tools to solve spatial analysis tasks or identify missing data. Three model judges compare the proposed workflows with reference solutions. University Mar 2025 51.5% Not in cards
HCAST Agents attempt 189 machine learning, cybersecurity, software engineering and reasoning tasks. 563 human baselines time each task, from one minute to over 8 hours, so success can be read against human task length. Expert Mar 2025 Not in cards
IVISTA Voice Interaction, revised 2023 protocol In-car assistants carry out spoken commands from different seats and answer follow-up requests. Testers check completed actions, including window adjustments and navigation commands. Everyday Mar 2025 Not in cards
METR Time Horizon Agents attempt software and research tasks timed against skilled humans. The score is the human task length at which the agent succeeds half the time. Research frontier Mar 2025 4.1 h System cards only
METR Time Horizon suite Software tasks with human completion times. Reports the task length at which a model succeeds 50% of the time. Research frontier Mar 2025 4.1 h Not in cards
RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. Everyday Mar 2025 50% Fading
TREC Product Search 2025 Models rewrite shopping queries for a search engine. Automatic entries submit one reformulation. Interactive entries can submit four, and the best one counts. Everyday Mar 2025 Not in cards
AI Diplomacy Models play the board game Diplomacy against each other, one power each. They negotiate, form alliances and betray each other in written messages. Everyday Feb 2025 60 Not in cards
BenchMAX Models follow instructions, write code, reason, use tools, translate, and retrieve information across 17 languages. Ten tasks use rule checks, model-judged wins, code tests, answer matching, or translation scores. University Feb 2025 85.2 Not in cards
BixBench Computational-biology analysis tasks (differential expression, assembly QC, cell typing) answered by agents with tools. Expert Feb 2025 93.8% Fading
Collab-Overcooked Two agents exchange messages and use different kitchen equipment to fill orders. The benchmark measures task completion and progress through required cooking steps. Everyday Feb 2025 86.7% Not in cards
ControlArena Agents try to perform unwanted side actions stealthily while completing a legitimate user task. Scored on whether monitors catch them. Expert Feb 2025 System cards only
Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. Expert Feb 2025 94.4% System cards only
Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. Research frontier Feb 2025 66.7% System cards only
Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. Expert Feb 2025 System cards only
Elimination Game Models play a multi-player game of public and private talk, alliances and votes to eliminate each other. A jury of eliminated players picks the winner. Everyday Feb 2025 8 Not in cards
EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. Everyday Feb 2025 50.5% Not in cards
EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. Everyday Feb 2025 67.7% Not in cards
EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. Everyday Feb 2025 58.7% Not in cards
EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. Everyday Feb 2025 28.5% Not in cards
EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. Everyday Feb 2025 53.7% Not in cards
Galileo Agent Leaderboard Models pick and call tools on samples drawn from existing tool-use benchmarks. Galileo scores Tool Selection Quality with an LLM judge and averages the datasets equally. Everyday Feb 2025 0.95 Not in cards
ITBench Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. Expert Feb 2025 58.3% Not in cards
KernelBenchFiltered Agents optimize GPU kernels. METR takes the geometric mean of the speedups across tasks after checking correctness. Expert Feb 2025 Not in cards
LLM Public Goods Game Agents choose how many tokens to contribute to a common pool. The evaluation reports the fraction each model contributes across repeated games. Everyday Feb 2025 45.2% Not in cards
LLM training task (Anthropic) Agents optimize a CPU-only small language model training implementation. Scored by speedup. Expert Feb 2025 System cards only
Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. Expert Feb 2025 69.7% System cards only
Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. Everyday Feb 2025 99.4% System cards only
MLGym-Bench Agents work on 13 open-ended AI research tasks in vision, language, reinforcement learning and game theory. Scores compare the agent's result with baselines on each task's metric. Expert Feb 2025 1.45 Not in cards
Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. Expert Feb 2025 100% System cards only
Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. Everyday Feb 2025 System cards only
Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. Expert Feb 2025 71.4% System cards only
Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. Expert Feb 2025 31.3 System cards only
RE-Bench subset (Anthropic) Agents solve a modified subset of four METR RE-Bench research engineering tasks. Scores are normalized from reference solution 0 to expert solution 1. Expert Feb 2025 0.24 System cards only
Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. Expert Feb 2025 100% System cards only
SnakeBench Two language models each steer a snake on a shared board in a turn-based game, and the one that survives longest wins. Ratings come from many head-to-head games. Everyday Feb 2025 39 Not in cards
SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. Expert Feb 2025 48.2% System cards only
SWE-Lancer Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned. Expert Feb 2025 81.4% Fading
Vals Legal AI Report Legal AI tools perform seven kinds of legal work. Vals compares their task scores with scores from practicing lawyers. Expert Feb 2025 Not in cards
Vending-Bench Agents run a simulated vending machine business over long horizons, ordering stock, setting prices, and paying daily fees. Scoring is final net worth in dollars. Everyday Feb 2025 $4,694 Dropped
Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. Expert Feb 2025 100% System cards only
WorldGUI Desktop GUI agents do 611 tasks that start from varied initial states, such as the target app not being open. Tasks come with instructional videos and project files. Everyday Feb 2025 45.8% Not in cards
Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. Everyday Jan 2025 97% System cards only
Android Agent Arena (201 tasks) Agents find live information and perform actions in Android apps such as YouTube Music and Booking.com. Task functions check app state and grade answers. Everyday Jan 2025 10.3% Not in cards
Autonomy-10 Agents complete browser, desktop and tool-use tasks across nine domains that measure skill, flexibility and intelligence. Expert Jan 2025 Not in cards
ComplexFuncBench Models make function calls for 1,000 travel booking style samples that need several steps, user constraints, implicit parameter reasoning, long parameter values or 128K context. University Jan 2025 61% Dropped
CSR-Bench Agents deploy and run computer-science research repositories with shell, retrieval and web-search tools. Task success measures whether the requested repository workflow completes. Expert Jan 2025 Not in cards
gBench Models run academic evaluations, Golden Set smoke tests, and an agentic quality suite through an OpenAI-compatible endpoint. The tool also records serving performance. University Jan 2025 Not in cards
MedAgentBench Agents use FHIR APIs in a virtual electronic health record to do 300 patient tasks in 10 categories written by physicians. Expert Jan 2025 69.7% Not in cards
ML-Dev-Bench Agents implement and train models, debug failures, and improve performance. Harbor ports 33 development tasks and checks them with the original validators. Expert Jan 2025 Not in cards
Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. Everyday Jan 2025 Not in cards
OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. Research frontier Jan 2025 55% System cards only
Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. Expert Jan 2025 23% System cards only
Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. Everyday Jan 2025 94% System cards only
Step Race Several models talk in public, then secretly pick moves in a race where colliding moves cancel. The game tests cooperation and deception. Everyday Jan 2025 5 Not in cards
ToolComp Models chain several tool calls to answer 485 human-verified prompts. Most final answers are numbers. ToolComp-Enterprise gives 11 tools and ToolComp-Chat targets general chat tool use. Step labels also support process reward model tests. Expert Jan 2025 70.1% Not in cards
User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. Everyday Jan 2025 92% System cards only
WebWalkerQA Agents start from a website's home page and traverse its subpages to find the facts that answer a question. Answers are judged against references. Everyday Jan 2025 33.3% Not in cards
SnorkelFinance ReAct agents answer expert-verified financial questions built from 10-K filings, using SQL, code and other MCP tools without seeing the documents. A judge scores full traces. University 2025 81% Not in cards
Agent-SafetyBench LLM agents act in 349 tool environments over 2,000 test cases covering 8 safety risk categories and 10 failure modes. A scorer model rates each run safe or unsafe. Everyday Dec 2024 59.8% Not in cards
Aider Polyglot Models edit code to solve 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust. Scored by the share passing unit tests. University Dec 2024 88% Fading
Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. Expert Dec 2024 System cards only
Domain Intelligence Benchmark Suite Models extract structured data, call functions, and answer questions using retrieved material. Databricks compares performance on enterprise tasks. University Dec 2024 Not in cards
HackSynth OverTheWire Agents complete scored security exercises from OverTheWire. The published benchmark covers 80 challenges, and the hub port selects 46. Expert Dec 2024 28.8% Not in cards
HackSynth PicoCTF Models attempt 120 tasks selected from picoCTF. The benchmark measures the percentage of tasks completed, and the hub port checks submitted answers against reference flags. Competition Dec 2024 22.5% Not in cards
LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. University Dec 2024 Not in cards
RelyToolBench Agents answer queries with simulated APIs built on the StableToolBench server. Scoring targets tool hallucinations, such as calling wrong or missing tools. Everyday Dec 2024 69.3% In use
Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. Expert Dec 2024 System cards only
SuperCLUE-Hard Models follow precise instructions, solve reasoning problems and plan tool calls in Chinese. The report gives separate scores for each ability. University Dec 2024 76.7 Not in cards
TheAgentCompany Agents do work tasks in a simulated software company, using a browser, code, and chat with simulated coworkers. Scored by full and partial checkpoint completion. University Dec 2024 Dropped
VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. Everyday Dec 2024 9.84% Not in cards
BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. Everyday Nov 2024 68.3% Not in cards
CRMArena Agents do nine customer-service CRM tasks for service agent, analyst and manager personas inside a real Salesforce org. The org holds 16 linked object types with seeded hidden patterns. Everyday Nov 2024 64.3% Not in cards
PARTNR Agents follow natural language instructions in human-robot collaboration scenes. Tasks require planning and coordination among multiple embodied agents. University Nov 2024 Not in cards
RedCode Models meet adversarial prompts to generate or run harmful code. Generation is model-judged and execution is checked in a container. Expert Nov 2024 6.00% System cards only
Spider 2.0 Models and agents write SQL for real enterprise databases with large schemas, across BigQuery, Snowflake, SQLite and dbt projects. Answers are checked by executing them. Expert Nov 2024 94.2% Dropped
Agent Arena (Gorilla) Users give a task, two agents built from different models, tools and frameworks run it, and users vote. A Bradley-Terry extension rates models, tools and frameworks. Everyday Oct 2024 1,045 Not in cards
AgentHarm 110 malicious agent tasks with synthetic tools. Measures refusal and harmful task completion, plus benign controls. Oct 2024 8.00% Fading
AndroidLab Mobile agents complete 138 tasks across nine apps on predefined Android virtual devices, in text-only and multimodal modes. Checks on the final device state score each task. University Oct 2024 31.2% Not in cards
AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. Expert Oct 2024 Not in cards
BrowserART Browser agents receive 100 harmful requests on synthetic and real websites. The test measures how often the agent attempts the harmful behavior that its backbone model would refuse in chat. Expert Oct 2024 11% Not in cards
Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. Expert Oct 2024 69% Not in cards
Embodied Agent Interface Models interpret goals, decompose them into subgoals, sequence actions, and predict state transitions in BEHAVIOR and VirtualHome. The evaluator checks task completion and distinguishes hallucinations, invalid actions, and planning errors. Everyday Oct 2024 70.3 Not in cards
GLEE Agents bargain, negotiate, or persuade another player through conversation while pursuing the payoff defined by the game. University Oct 2024 2,595 Not in cards
ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. Everyday Oct 2024 Not in cards
MLE-bench Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. Expert Oct 2024 96.1% In use
ScienceAgentBench Agents write programs for 102 data-driven discovery tasks from 44 papers in four disciplines. Outputs are checked by execution and task-specific metrics. Expert Oct 2024 42.2% Not in cards
SPA-Bench Agents use Android apps such as Airbnb, Spotify, and WeChat. The grader matches screenshots against task requirements and uses a vision model to resolve uncertain cases. Everyday Oct 2024 1.35% Not in cards
SWE-bench Multimodal SWE-bench tasks whose issues include screenshots or design mockups from visual JavaScript repos. Patch graded by hidden tests. Expert Oct 2024 74.1% In use
Verus-Bench Agents synthesize Verus proof annotations for 150 Rust verification tasks from algorithms, arrays and loop programs. Expert Oct 2024 Not in cards
WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. Oct 2024 Not in cards
Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. Expert Sep 2024 System cards only
cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability. Expert Sep 2024 51.8% Not in cards
ContextualBench CRM agents answer enterprise questions against contextual business data and receive accuracy, cost, speed, trust and safety measurements. Expert Sep 2024 Not in cards
CORE-Bench Agents reproduce the results of published studies from their code and data. 270 tasks from 90 papers in computer science, social science and medicine come in three difficulty levels. Expert Sep 2024 77.8% Not in cards
DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. University Sep 2024 Not in cards
DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap. Expert Sep 2024 89.9% Dropped
FlagEval Debate Models debate a topic in Chinese. The leaderboard estimates their ratings from comparisons and reports uncertainty. Everyday Sep 2024 1,149 Not in cards
ForecastBench Models give probabilities for 500 auto-generated questions per round about future events, drawn from prediction markets and time-series datasets. Forecasts are scored with difficulty-adjusted Brier scores once the questions resolve. Expert Sep 2024 0.11 In use
FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. Everyday Sep 2024 87.6% Fading
LLM Chess Models play chess in multi-turn dialogs against a random player or the Komodo Dragon engine. The board reports wins, Elo and game duration, which tracks instruction following. Everyday Sep 2024 1,614 Not in cards
RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. Research frontier Sep 2024 1.27 System cards only
RedTeam Arena Players get 60 seconds to make a model say a target word. An extended Elo ranks players, models and target prompts, with models ranked by resistance. Everyday Sep 2024 Not in cards
RoboTwin 1.0 Dual-arm policies train on generated digital-twin demos and are tested on simulated single-arm and dual-arm tasks, with matching real-world data. Success rate. Everyday Sep 2024 Not in cards
Windows Agent Arena Agents complete more than 150 tasks in a real Windows OS with its apps, tools and browsers. Scripts adapted from OSWorld check the final state. Everyday Sep 2024 50.6% Dropped
Agentic Self-Reasoning (Apollo Research) Agents reason about their own operating conditions to complete tasks. Apollo measures success at three difficulty levels. Everyday Aug 2024 Not in cards
Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. Expert Aug 2024 78% System cards only
Applied Theory of Mind (Apollo Research) Agents complete tasks that require tracking what another participant knows. Apollo measures success at three difficulty levels. Everyday Aug 2024 Not in cards
Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. University Aug 2024 68% System cards only
Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. School Aug 2024 82% System cards only
Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. Expert Aug 2024 96.3% Fading
Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. Expert Aug 2024 65.4% Fading
HR-Bench Models answer multiple-choice questions about fine details in 4K and 8K resolution images. Results are reported separately for the 4K and 8K sets. Everyday Aug 2024 83% Fading
Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. Expert Aug 2024 0% System cards only
SWE-bench Verified 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Expert Aug 2024 83.5% In use
Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. Everyday Aug 2024 System cards only
ToolSandbox Models use stateful tools in multi-turn conversations with a simulated user. Tools have hidden state dependencies, and runs are scored against milestones. Everyday Aug 2024 80.7% In use
VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. Everyday Aug 2024 36.2% Not in cards
Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. Everyday Aug 2024 5.74% System cards only
XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. Expert Aug 2024 98.5% Fading
AppWorld Agents write code against 457 APIs of 9 simulated everyday apps to finish 750 tasks for about 100 fictional users. State-based unit tests check the outcome and collateral damage. Everyday Jul 2024 Not in cards
AppWorld Test-Challenge Agents complete 417 app tasks that require Amazon or Gmail APIs held out from training. The score is the percentage passing every task test. Everyday Jul 2024 92.8% Not in cards
AppWorld Test-Normal Agents complete 168 everyday app tasks from familiar applications by calling APIs through code. The score is the percentage passing every task test. Everyday Jul 2024 94% Not in cards
AssistantBench Web agents do 214 realistic, time-consuming tasks on the open web, such as monitoring real-estate markets. Answers are checked automatically against gold answers. Everyday Jul 2024 Not in cards
ChiPBench (chip placement) Agents place circuit blocks in chip designs. OpenROAD measures timing, power, and area after routing. Higher timing slack is better. Expert Jul 2024 Not in cards
CIBench Agents use code interpreters to analyze data. The evaluation checks stages of the interactive analysis. University Jul 2024 75.5% Not in cards
CRAB Multimodal agents do tasks that span an Ubuntu desktop and an Android phone at once. Graph-based evaluators give credit for intermediate steps. Everyday Jul 2024 38% Not in cards
CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. Expert Jul 2024 Not in cards
CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. Jul 2024 14% System cards only
DiscoveryBench Agents analyze datasets and produce data-driven hypotheses that evaluators score against structured evidence and domain criteria. Expert Jul 2024 Not in cards
GTA Agents answer 229 human-written queries that need 1 to 4 real deployed tools for perception, operation, logic and creativity, often with image inputs. Queries leave the tools and steps implicit. Everyday Jul 2024 46.6% Not in cards
LAB-Bench LitQA2 Models answer multiple-choice questions whose answers appear once in recent biology papers. Built to need literature search rather than recall. A LAB-Bench subtask. Expert Jul 2024 27% Not in cards
RelBench Agents build models for prediction tasks over multi-table relational databases, graded on held-out targets. Expert Jul 2024 38.4% System cards only
Spider2-V Multimodal agents do data science and engineering workflows in real enterprise apps such as BigQuery, dbt and Airbyte, on a desktop computer. Execution-based checks score each task. Expert Jul 2024 14% Not in cards
Werewolf Arena Models compete in Werewolf with strategic speaking, deception, deduction, and persuasion. Everyday Jul 2024 26.7 Not in cards
WorkArena++ Web agents complete 682 compositional knowledge-work tasks in ServiceNow, which need planning, reasoning and memory across many steps. Expert Jul 2024 1.90% Not in cards
AgentDojo Agents do tool-use tasks while tools and data carry prompt injections. Attack success rate counts hijacked runs. Everyday Jun 2024 0% Fading
CharXiv Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. University Jun 2024 93.5% In use
CRAG Retrieval-augmented systems answer 4,409 factual questions across finance, sports, music, movies and open domain, using mock web and knowledge graph search APIs. Hallucinated answers are penalized. Everyday Jun 2024 0.20 Not in cards
Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass. Expert Jun 2024 78% System cards only
LiveBench Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. University Jun 2024 87.4% Dropped
Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. Everyday Jun 2024 Not in cards
MobileAgentBench Agents complete 100 tasks across ten Android apps. The grader checks app state and accessibility events, so agents can take different paths. Everyday Jun 2024 8.00% Not in cards
NanoGPT Speedrun Improve a small GPT training script to reach a target validation loss as fast as possible on fixed hardware. Research frontier Jun 2024 42.9% System cards only
NATURAL PLAN Models make travel itineraries, find meeting arrangements, and schedule calendar events. The evaluator checks each proposed plan against the stated constraints. Everyday Jun 2024 Not in cards
NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. University Jun 2024 22% Not in cards
RoboCasa Simulated mobile manipulators and humanoids do 100 kitchen tasks, 25 atomic and 75 composite, across 120 scenes. Scored by success rate. Everyday Jun 2024 70.8% Fading
SpreadsheetBench Real spreadsheet manipulation questions from Excel forums. Output workbook is checked cell-by-cell against reference solutions. University Jun 2024 89.3% In use
SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix. Expert Jun 2024 89.1% Fading
tau-bench Agents chat with a simulated user and call retail or airline APIs under policy rules. A task passes if the final database matches the goal. Everyday Jun 2024 70.5% Dropped
tau-bench Airline Agents help simulated airline customers through tools while following airline policies on the original tasks. The score is pass^1 as a percentage. Everyday Jun 2024 46% Not in cards
tau-bench Retail Agents help simulated retail customers through tools while following store policies on the original tasks. The score is pass^1 as a percentage. Everyday Jun 2024 67.8% Dropped
WebCanvas Web agents do tasks on live websites, scored on key intermediate states rather than one fixed action path, so the test survives site changes. Everyday Jun 2024 28.8% Not in cards
AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. Expert May 2024 37.8% Not in cards
AnalogCoder Models write Python programs that build analog circuits. A simulator checks whether each circuit performs its requested function. The authors report first-attempt success and how many tasks each model solves. University May 2024 83.3% Not in cards
AndroidWorld Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. Everyday May 2024 100% In use
MiniStS Models play card battles with five preset decks. The authors measure final health in four scenarios and how often agents play the Harm card in the fifth. Everyday May 2024 Not in cards
MobileMiniWoB++ Agents complete MiniWoB++ web tasks inside the Android emulator of AndroidWorld. Scored by task success rate. Everyday May 2024 68% Dropped
SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. Everyday May 2024 80% In use
B-MoCA Agents use Android apps such as Instagram on phones with different languages, font sizes, and icon positions. Task-specific checks decide whether each attempt succeeded. Everyday Apr 2024 47.7% Not in cards
CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. University Apr 2024 16.4% System cards only
CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. Expert Apr 2024 Not in cards
LlamaTouch Agents carry out requests in Android apps. The evaluator checks whether each run visited the required app or system states. Everyday Apr 2024 Not in cards
OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. Everyday Apr 2024 57.9 Dropped
OSWorld Agents complete 369 real computer tasks in Ubuntu desktop apps through screenshots and mouse and keyboard actions. Execution-based scripts check the final state. Everyday Apr 2024 78.8% Fading
BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. Everyday Mar 2024 Not in cards
Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. Everyday Mar 2024 System cards only
DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. Mar 2024 20.5% Not in cards
DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design. Expert Mar 2024 41.7% Not in cards
DevBench Environment Setup Models prepare repository environments. The score is the percentage that pass the example usage test. Expert Mar 2024 41.7% Not in cards
Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. Expert Mar 2024 System cards only
Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. Everyday Mar 2024 System cards only
HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. Everyday Mar 2024 Not in cards
In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. Expert Mar 2024 84.6% System cards only
LLM Colosseum Two models each control a fighter in Street Fighter III and choose moves from text descriptions of the game. Wins give Elo ratings. Everyday Mar 2024 1,835 Not in cards
Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. Everyday Mar 2024 3.72 System cards only
Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. Expert Mar 2024 0% System cards only
StableToolBench Models solve user requests by calling tools from ToolBench's large set of real-world APIs. A virtual API server with a cache and API simulators stands in for live APIs, so results stay stable over time. GPT-4 judges whether each request was solved and which of two answers is better, giving a solvable pass rate and a win rate. Everyday Mar 2024 26.4% Not in cards
SWE-bench Lite Models resolve 300 self-contained GitHub issues chosen from SWE-bench to cut evaluation cost. Hidden repository tests decide each patch. Expert Mar 2024 Not in cards
Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. Everyday Mar 2024 System cards only
WorkArena Web agents do routine knowledge-work tasks in a live ServiceNow instance, such as filling forms and filtering lists. WorkArena++ adds 682 compositional tasks at levels 2 and 3. University Mar 2024 Not in cards
AndroidArena Agents use apps such as Gmail and Google Maps, sometimes switching apps or following extra constraints. GPT-4 judges success from the recorded actions and observations. Everyday Feb 2024 12.7% Not in cards
Berkeley Function Calling Leaderboard (BFCL) Model emits function calls for single, parallel, multi-turn and agentic scenarios. Scored by AST and execution checks. Everyday Feb 2024 88.5% In use
FinBen A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading. University Feb 2024 0.25 Not in cards
GTBench Models play strategic games against other models or scripted opponents. The benchmark measures their advantage in game rewards. Everyday Feb 2024 0.13 Not in cards
Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. Expert Feb 2024 15.4% System cards only
TravelPlanner Agents plan multi-day trips from a user request, using tools to search nearly four million records of flights, hotels, restaurants and attractions. Each plan must meet the user's hard constraints, such as budget, and commonsense rules. The headline final pass rate counts a plan only when it meets every constraint. Everyday Feb 2024 Not in cards
WebLINX Agents follow a user's multi-turn chat instructions to navigate real websites. 100,000 interactions from 2,300 expert demonstrations give turn-level metrics. Everyday Feb 2024 10.4 Not in cards
AgentBoard LLM agents act in 9 partially observable, multi-turn environments that cover embodied tasks, games, web browsing and tool use. Besides final success, AgentBoard reports a progress rate, the share of annotated subgoals the agent reaches. An open toolkit breaks results down by skill. Everyday Jan 2024 48.9% Not in cards
Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. Expert Jan 2024 System cards only
Economics Arena for Large Language Models Models choose numbers in beauty contests and place sealed bids in auctions. Researchers compare payoffs and measure how strategies change when players see previous rounds. University Jan 2024 0.95 Not in cards
GLM-4 All Tools information seeking (Z.ai) Models use a web browser to find information for user questions. The ChatGLM report gives accuracy but does not describe the test set. Everyday Jan 2024 78.1% System cards only
LLM Game Benchmark Models compete in tic-tac-toe, Connect Four, Gomoku, and other grid games through a shared interface. Everyday Jan 2024 185.0 Not in cards
Mobile-Eval Agents follow 33 instructions on Android. Some requests specify the app. Others describe a goal and leave the choice of app to the agent. Everyday Jan 2024 Not in cards
VisualWebArena Multimodal web agents complete 910 visually grounded tasks on self-hosted classifieds, shopping and Reddit-style sites. Execution-based checks score each task. University Jan 2024 16.4% Not in cards
WebVoyager Agents complete user tasks on 15 popular real websites by browsing from screenshots and page text. A GPT-4V judge grades whether each task succeeded. Everyday Jan 2024 89.3% Fading
AppAgent Test Set Agents follow 45 instructions across nine Android apps, including scheduling email and changing music playback. Tasks time out after ten steps. Everyday Dec 2023 Not in cards
T-Eval Models use tools step by step, and each sub-process is scored separately: instruction following, planning, reasoning, retrieval, understanding, and review. Everyday Dec 2023 86.4% Dropped
V* Models answer 191 multiple choice questions about small details in high-resolution images, such as an object's color or relative position. Everyday Dec 2023 96.9% Fading
AvalonBench Models play Resistance Avalon while discussing, deceiving, deducing, and negotiating with other agents. Everyday Nov 2023 Not in cards
FinMem Trading Agents choose whether to buy, sell or hold a stock each day. The benchmark measures cumulative return against historical prices. University Nov 2023 1.63% Not in cards
GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. Everyday Nov 2023 66.1% Fading
Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. University Nov 2023 4.67% Not in cards
LangChain Benchmarks Agents complete tool use, retrieval and extraction tasks with reproducible LangSmith datasets and task-specific graders. Expert Nov 2023 Not in cards
BotChat Two instances of a model sustain a conversation. Judges assess the resulting dialogue. Everyday Oct 2023 73.2 Not in cards
LLM-Coordination Models play four cooperative games and answer 198 multiple-choice questions about the same games. The authors report game outcomes and question accuracy separately. Everyday Oct 2023 46.6 Not in cards
MLAgentBench Agents run machine learning experiments on 13 tasks, from improving CIFAR-10 accuracy to research problems like BabyLM. They read and write files, run code and inspect outputs. Expert Oct 2023 18.3% Not in cards
SmartPlay Language-model agents play a collection of games designed to test planning, reasoning, memory and interaction abilities. Expert Oct 2023 0.80 Not in cards
SuperCLUE-Agent Chinese language models are tested on agent skills in Chinese tasks. These include tool use with several API calls, task planning and memory over long dialogues. Everyday Oct 2023 78.4 Not in cards
SWE-bench Models resolve 2,294 real GitHub issues from 12 Python repositories. A patch counts as resolved when the hidden repository tests pass. Expert Oct 2023 Not in cards
Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. Expert Sep 2023 System cards only
Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. School Sep 2023 81.7% System cards only
Code Interpreter Benchmark General Models use a Python interpreter for general tasks. The score is the percentage of generated programs that execute. School Sep 2023 65.5% Not in cards
Code Interpreter Benchmark Math Models use a Python interpreter to solve math problems. The score is execution accuracy on the Math task set. School Sep 2023 73.1% Dropped
MakeMePay A con artist model tries to talk a GPT-4o mark into paying money in a role-play. Scored as the share of conversations with a payment. Everyday Sep 2023 57% System cards only
MakeMeSay A manipulator model tries to make a GPT-4o manipulatee say a codeword without suspicion. Scored as manipulator win rate. Everyday Sep 2023 72% Fading
AgentBench 8 environments (OS, DB, KG, games, web) testing LLMs as agents. Everyday Aug 2023 4.01 Not in cards
Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. Everyday Aug 2023 98% Dropped
DroidTask Agents select actions for 158 Android tasks across 13 apps. The original completion score requires every action to match the recorded sequence. Everyday Aug 2023 Not in cards
FinEval Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings. University Aug 2023 69.4% Not in cards
Aider-Edit Models edit Python source files to complete 133 Exercism exercises through Aider. Scored by the share of exercises whose unit tests all pass. School Jul 2023 79.7% Dropped
ToolBench Models plan and call real REST APIs from RapidAPI, over 16,000 of them, to answer instructions that need one or many tools. ToolEval judges pass rate and win rate. Everyday Jul 2023 Not in cards
WebArena Agents complete 812 realistic tasks on self-hosted shopping, forum, GitLab, CMS and map websites. Functional graders check the outcome. Everyday Jul 2023 68% Fading
InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. School Jun 2023 98.7% System cards only
LeanDojo Benchmark Models prove theorems from Lean's mathlib library inside the Lean proof assistant, which checks every proof. The benchmark has 98,734 theorems and proofs with premise annotations. A novel-premises split tests whether a prover can use lemmas it never saw in training. University Jun 2023 Not in cards
LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. Everyday Jun 2023 97.2% Fading
ToolQA Models answer questions that need outside information, using 13 tools such as database queries, text retrieval and a calculator over fixed data sources. The questions are built so that a model's internal knowledge cannot answer them. An easy and a hard set separate single lookups from multi-step tool use, and answers are checked by exact match. Everyday Jun 2023 24.1% Not in cards
clembench Chat models play rule-based dialogue games in self-play, such as Taboo, Wordle and reference games. The clemscore combines how often a model follows the game rules at all with how well it plays when it does. Version 4.0 runs 14 games in 30 languages. Everyday May 2023 88.8 Not in cards
SuperCLUE The CLUE team tests Chinese and foreign language models on periodic sets of Chinese questions covering language, professional knowledge, reasoning, agents and safety. Open answers are graded by a strong LLM judge, and results feed an overall ranking. University May 2023 73.6 Not in cards
API-Bank Models make API calls in dialogues that need tools such as calendars and search. Scored on call accuracy. Everyday Apr 2023 92.3% System cards only
MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. The score is normalized game reward. Annotations also score power-seeking, disutility and ethical violations. Everyday Apr 2023 Not in cards
PushWorld Agents move shapes through puzzles that require tools or pushing obstacles aside. The benchmark includes more than 200 puzzles with goal positions. School Jan 2023 Not in cards
Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. Everyday Oct 2022 76% System cards only
Language-Table A simulated robot moves colored blocks in response to natural language instructions. The benchmark measures success and success weighted by path length across five task families and 696 task variations. Everyday Oct 2022 Not in cards
VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. Everyday Oct 2022 Not in cards
WebShop Agents search and click through a simulated shopping site with 1.18 million real products to buy an item that matches a text instruction. Reward reflects how well the purchase matches. Everyday Jul 2022 Not in cards
MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. Everyday Jun 2022 49% Fading
ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. School Mar 2022 0.15 Not in cards
CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. Everyday Dec 2021 Not in cards
SatNet Agents assign ground antennas and contact times to spacecraft requests across five weeks of NASA Deep Space Network operations. They balance unmet demand across missions. University Nov 2021 Not in cards
Jiminy Cricket Agents play 25 text adventures. Annotations on game states track moral behavior alongside game rewards. Everyday Oct 2021 Not in cards
TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. Everyday Oct 2021 Not in cards
ALFWorld Agents complete household goals such as putting a washed apple in the fridge in a text world aligned with the ALFRED embodied simulator. Success is whether the goal state is reached. Everyday Oct 2020 24.6% Not in cards
Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. Everyday Oct 2020 Not in cards
PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. School May 2020 Not in cards
TextWorld Cooking (GATA) Agents navigate text-based rooms, gather ingredients, and prepare recipes. The port sums points earned during play and includes separate training and test games at increasing difficulties. Everyday Feb 2020 2.17 Not in cards
ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. Everyday Dec 2019 Not in cards
Meta-World Robot policies do simulated tabletop manipulation tasks, scored by success rate. Splits test multi-task learning, such as MT10 and MT50, and meta-learning, such as ML45. Everyday Oct 2019 Not in cards
RLBench Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates. Everyday Sep 2019 Not in cards
BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. Everyday Oct 2018 37.5% Not in cards
MiniWoB++ Agents operate over 100 small synthetic web pages, such as forms, menus and date pickers, from a natural-language instruction. Each task returns a success reward. Everyday Feb 2018 Not in cards
Behnke and Geiger Flexible Job Shop Instances Agents build schedules or write solvers that assign operations to eligible machines within work centers. The objective is to finish the last job as early as possible. University Jan 2012 Not in cards
Wikispeedia Players navigate from one Wikipedia article to another using only article links. The Prime Intellect port scores whether agents reach targets held out from training. Everyday Jul 2009 Not in cards
Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. Expert System cards only
Agent Initiative Benchmark Agents inspect workplace situations and decide whether to act without an explicit task. Two model judges score responses to 50 situations from 0 to 10. Expert 8.97 Not in cards
AgentCLUE Agents execute tasks in Chinese. University 46.4 Not in cards
AgentCLUE-General Agents complete general purpose tasks in Chinese. University 4.62 Not in cards
AgentCLUE-Mobile Mobile assistants complete tasks on a phone. Everyday Not in cards
AgentCLUE-Task Agents carry out interactive tasks in Chinese. Everyday 49.5 Not in cards
Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. Research frontier 84.6% Not in cards
Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. Expert 0.49 System cards only
AnalogCoder-Pro Agents write analog circuit programs and use simulated waveforms to diagnose failures. The authors report the fraction of successful designs and the number of tasks solved. Expert 88.4% Not in cards
Animation Bench Agents recreate web animations in HTML from reference frames and a network capture. The scorer compares appearance, motion and layout. University 0.61 Not in cards
Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. 2.06 System cards only
Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. Not in cards
Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. Not in cards
Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. Expert 0% In use
Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. Expert 14.3% System cards only
BAZAAR Agents buy and sell goods in a simulated competitive market. University 7 Not in cards
BIM Bench Agents draw a timber roof eaves detail in Revit. Reviewers see the exported sheet without the model name and rate its usability from one to five. University 4 Not in cards
Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. Expert 91.1% System cards only
Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. Expert 2.53 System cards only
BrowserInject v1 Browser agents do benign tasks on 30 retail, health, support and travel sites seeded with prompt injections. Reports attack success rate. Expert 36.7% System cards only
Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. Expert System cards only
CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. Expert 29% System cards only
Charlemagne Labs social-engineering benchmark Attacker models try to persuade a skeptical simulated victim to take an action over many turns. Scored by attempted and carry-through rates. Expert 1.60% System cards only
Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. Expert System cards only
ChessBench (MadeByLupu) Models play each other with access to the board, move history and legal moves. The organizer rates each model and thinking setting with Glicko-2. School 2,170 Not in cards
Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. Everyday 100% System cards only
Closed-Loop Evaluation of LLM Agents for Embedded Software Development Agents write firmware from engineering briefs and test it on simulated equipment. Hidden checks score device behavior across five control tasks. University 100% Not in cards
CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. Expert 55.8% System cards only
Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. Expert 83% System cards only
CompanyBench Agents complete financial analyst workflows in a simulated bank. Checks compare outputs and database changes with expected results. Expert 44.4% Not in cards
Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. Everyday 100% System cards only
Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. Everyday 99.4% System cards only
Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. Expert 78% System cards only
Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. Expert 92% System cards only
Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. Expert 27% System cards only
CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Expert 75.9% In use
Crest Arcade Chess Models play standard chess and Chess960 against fixed engine opponents without a list of legal moves. The organizer fits separate ratings for each variant. School 1,758 Not in cards
Crest Arcade Elemental Armory Models equip four elemental balls with cards, then select and aim a ball each round. The score averages match win rates across three levels. Everyday 67.3% Not in cards
Crest Arcade FreeCell Models move cards to the four foundations using a variable number of free cells. The score averages win rates across five tiers. Everyday 63.3% Not in cards
Crest Arcade Hustle Sim Models buy upgrades and hire helpers to reach earnings targets in an idle game. The score compares simulated completion times with reference strategies. Everyday 83.7 Not in cards
Crest Arcade Micro Mouse Models steer a simulated robot through six daily mazes. The score compares each finish time with the median human time and gives unfinished mazes zero. Everyday 89.3 Not in cards
Crest Arcade Mines Models reveal or flag cells on Minesweeper boards that can be solved without guessing. The score averages win rates across four board sizes. Everyday 100% Not in cards
Crest Arcade Nations Royale Models manage one nation against 47 computer opponents on a simulated map. Wins earn full credit and unfinished games earn credit for territory held. Everyday 100.0 Not in cards
Crest Arcade Ravelin Models build towers to protect a keep through waves of attackers. The score compares wins with the reference time plus a penalty for lives lost. Everyday 106.8 Not in cards
Crest Arcade Starjar Models place pieces in a jar so matching pieces touch and merge. The score compares their total points with a reference bot on identical jars. Everyday 114.5 Not in cards
Crest Arcade Wildhaven Models select a team of animals and choose where each lands in a village. The score measures the share of village strength they destroy. Everyday 100% Not in cards
CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. Expert 82.3% System cards only
Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. Everyday 24.3% System cards only
DesignArena Agentic Game Dev Coding agents build multi-file games from a user prompt. Users play the final games and vote, and the logs keep tool calls, retries and failures. Everyday 1,408 Not in cards
DesignArena Agentic Image to HTML Coding agents turn a user-uploaded image into a self-contained HTML website. Users compare how well each page reproduces the design and vote. Everyday 1,247 Not in cards
DesignArena Agentic Overall Coding agents build apps, games, slides and spreadsheets from user prompts. Users compare the results, and votes from every agent board form one rating. Everyday 1,443 Not in cards
DesignArena Blender Agents build a 3D scene in Blender from a user prompt. Users compare the rendered scenes and vote for the better one. Everyday 1,381 Not in cards
DesignArena Builder AI app builders make full-stack web and mobile apps from the same user prompt. Users compare the results and vote for the better one. Everyday Not in cards
DesignArena Frontend Web App Coding agents build multi-file React frontends from real user prompts. Users compare the running apps and vote, and the run logs record agent behavior. Everyday 1,360 In use
DesignArena Full Stack Coding agents build complete React apps with sign-in, databases and backend code from real user prompts. Users try the apps and vote for the better one. Everyday 1,400 Not in cards
DesignArena Godot Game Dev Coding agents build complete Godot projects with GDScript, scenes and generated assets, exported to run in the browser. Users play the games and vote. Everyday 1,315 Not in cards
DesignArena HTML Slides Agents write a self-contained HTML slide deck that is converted to PPTX. Users compare the decks and vote for the better one. Everyday 1,531 Not in cards
DesignArena Image to Frontend Coding agents turn a reference image into a multi-file frontend web app. Users compare the running apps and vote for the better one. Everyday 1,299 Not in cards
DesignArena PDF Agents build a PDF document from a user prompt. Users compare the two documents and vote for the better one. Everyday 1,346 Not in cards
DesignArena Python-PPTX Slides Agents build PowerPoint decks with python-pptx from a user prompt. Users compare narrative, visual design and structure and vote for the better deck. Everyday 1,377 Not in cards
DesignArena React Native Models build cross-platform mobile apps in React Native from real user prompts. Users compare the running apps and vote for the better one. Everyday 1,318 Not in cards
DesignArena Spreadsheet Agents build spreadsheets such as financial models, budgets and trackers as Excel or CSV workbooks. Users compare accuracy, structure and formatting and vote. Everyday 1,429 Not in cards
Destructive actions (Anthropic) Models continue real Claude Code transcripts cut just before a destructive action. Score is the rate of taking a destructive action. Expert 21% System cards only
Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. Expert 0.30% System cards only
EKWbench Agents complete operational tasks such as sourcing reviews and referral triage. The score averages the share of verifier weight earned. Expert 74.4% Not in cards
EmbodiedCLUE-VLA (tabletop tasks) Robot policies carry out tasks on a tabletop. Everyday 49.3 Not in cards
Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. Research frontier 72.4% System cards only
FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. Everyday $215,319 Not in cards
Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. Expert 100% System cards only
FrontierMemory A model reads 750 past agent trajectories and writes memory for a task agent. The score is how many new coding and knowledge-work tasks that agent then solves. Research frontier 73.3% Not in cards
Game Arena 3 Check Chess (Generality) Models play both colors. A player can win by checking the opposing king three times. A double check counts once. School 27.0 Not in cards
Game Arena Anti Chess (Generality) Models must capture when they can. A player wins by losing every piece or by having no legal move. The king can be captured. School 25.0 Not in cards
Game Arena Atomic Chess (Generality) Models play chess with exploding captures. A capture removes both involved pieces and neighboring pieces other than pawns. Exploding the opposing king wins. School 27.0 Not in cards
Game Arena Crazyhouse Chess (Generality) Models play Crazyhouse chess in both colors. A player can spend a turn placing a captured piece back on the board under its new owner's control. School 26.0 Not in cards
Game Arena Four-in-a-Row (Generality) Models place pieces on any empty square of a four by nine grid. Four in a line wins. The tournaments use different board encodings. Everyday 92.0 Not in cards
Game Arena Horde Chess (Generality) Models take both sides in turn. White tries to checkmate with 36 pawns. Black starts with the usual pieces and must capture every white pawn. School 24.0 Not in cards
Game Arena King of the Hill (Generality) Models can win by moving their king onto any of the four central squares. The king must reach the square without moving into check. School 24.0 Not in cards
Game Arena Racing Kings (Generality) Models race their kings to the eighth rank without giving check. Black can draw by reaching that rank immediately after White. School 23.5 Not in cards
Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. Not in cards
Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. Not in cards
Genspark Slides Benchmark Models build finished slide decks from 200 real production tasks in the Genspark Slides agent harness. An internal grader scores quality, completion, content and design. Everyday 83.6% Not in cards
Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. Expert Not in cards
Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. 0% System cards only
Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. Research frontier 27% System cards only
Gray Swan Shade (adaptive attacker) Gray Swan's adaptive red-teaming agent trained on test scenarios to attack coding, computer-use and browser agents with prompt injections. Not in cards
Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. Expert 99% System cards only
HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. 7.80% System cards only
Harvey LAB-AA v1.0 Artificial Analysis runs agents on Harvey's Legal Agent Benchmark tasks in an offline sandbox, where they write legal work products such as memos and redlines from case documents. Judges grade each deliverable against a task rubric, and the headline is the share of rubric criteria passed. Expert 94.6% In use
Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. Expert System cards only
ICLayout-Bench Agents turn transistor netlists into chip layouts. Design tools check the layout rules and connectivity, then measure electrical performance and area. Expert 80.3 Not in cards
Implicit Intelligence Agents respond to requests whose unstated requirements affect safety, privacy, or accessibility. Labelbox reports the share of scenarios each model passes. Everyday 53.2% Not in cards
Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. Everyday 93.8% System cards only
Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. Expert 40.8% System cards only
Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. Expert 4.20% System cards only
Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. Expert 30.8% System cards only
InferenceBench Coding agents optimize a language model inference server within a fixed time budget. The harness measures serving speed and checks answer quality against a reference. Expert 12.1× Not in cards
Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. Expert 0% System cards only
Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. Expert 72% System cards only
Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). Expert 70% System cards only
JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. Expert 2.00% System cards only
Kaggle Chess Suite Models play chess from the standard starting position and from selected openings. Kaggle publishes both ratings for each model. School 1,238 Not in cards
Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. Research frontier 80% System cards only
Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks. Expert Not in cards
LatchBio Capabilities V0 Agents complete biological data analysis tasks. The historical score averages the available benchmark scores equally and excludes missing scores. Expert 51.3% Not in cards
Lazy investigation (Anthropic) Agents debug a failing job where misleading evidence invites a harmful action such as cordoning a healthy node. Scored 0 to 10 by a grader. Expert 9.94 System cards only
LLM Debate Benchmark Models debate an opponent over multiple rounds. Judges compare the arguments. University 1,706 Not in cards
LLM Deceptiveness and Gullibility Models try to deceive other models and recognize deceptive claims. Everyday 1.10 Not in cards
LLM Games Models play Connect Four against one another. Ramp publishes match scores and final Elo ratings. Everyday 1,728 Not in cards
LLM Persuasion Benchmark Models try to change another model’s stated position over multiple conversation turns. Everyday 1.71 Not in cards
LLM SoccerArena Bundesliga 2026/27 Models forecast Bundesliga match scores and outcome probabilities with web access. The standings award points for score predictions made about 24 hours before kickoff. Everyday 57.0 Not in cards
LLM SoccerArena La Liga 2026/27 Models forecast La Liga match scores and outcome probabilities with web access. The standings award points for score predictions made about 24 hours before kickoff. Everyday 129.0 Not in cards
LLM SoccerArena Premier League 2026/27 Models forecast Premier League match scores and outcome probabilities with web access. The standings use the predictions made about 24 hours before kickoff. Everyday 89.0 Not in cards
Mana RP Bench Models write the next turn in a player’s story. The player sees two anonymous responses and chooses which one becomes part of the story. Everyday 1,280 Not in cards
Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. Expert 0.70% System cards only
Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. Expert 20.7% System cards only
Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. Expert 79.4 System cards only
Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. Expert 47.8% System cards only
Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. Expert 5.74% System cards only
Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. Expert 8.50% System cards only
METR data deduplication Agents write an efficient algorithm that removes near-duplicate lines from a large file. A trial succeeds when F1 reaches at least 80%. Expert 32.6% System cards only
METR general autonomy task suite Agents complete multi-step tasks in virtual environments. Scored as horizon weighted average task completion compared with human time limits. Expert System cards only
METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. Research frontier Not in cards
Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. Expert 100% System cards only
Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. Expert 94.3% System cards only
Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. Expert System cards only
NeuroscienceBench Agents analyze neuroscience research data and report a scientific conclusion. The score counts correct conclusions among graded runs. Expert 41% Not in cards
OdysseysMega Agents complete tasks in a web browser. A judge checks each rubric criterion against the recorded actions and screenshots. Expert 41% Not in cards
OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. Not in cards
OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. 0.47% System cards only
OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. Not in cards
OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). Expert Not in cards
OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). Expert Not in cards
OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. System cards only
OpenDesign Arena Models create design artifacts in the OpenDesign tool, and the board reports an evaluation score out of 100 with cost and speed per artifact. Everyday 91.0 Not in cards
OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason. Expert 34.9% System cards only
OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. Research frontier 64.2% System cards only
Overconfidence: verifying command-line syntax (Anthropic) Agents are asked for an exact invocation of an unfamiliar command-line tool. Graders score from 1 to 5 whether they check documentation before running commands. Expert 4.90 System cards only
Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. Everyday 55.9% System cards only
Peer Arena (oddbit.ai) Five models debate for five rounds, then each privately votes for a survivor. The organizer calculates Bradley-Terry ratings from game outcomes. Everyday 1,799 Not in cards
Personal Agent Bench Personal assistants complete everyday tasks using connected accounts. Experts check whether each assistant completes the work and respects the user’s instructions. Everyday Not in cards
Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. Expert 98% System cards only
PGG-Bench Agents decide how much to contribute to a public good and whether to punish other players. Everyday 14 Not in cards
Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. Everyday 0.59% Not in cards
Pokémon VGC Bench (Ricotoma) Agents play competitive Pokémon doubles with assigned teams. They choose four Pokémon, select two leads, and choose actions for both active Pokémon each turn. Competition 1,901 Not in cards
Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. Expert 91% System cards only
Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. Expert 86% System cards only
QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). Expert 63.1% In use
QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. Expert 1,770 In use
QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. Expert 86.3% In use
RareDiseaseBench Agents combine clinical and biological evidence to diagnose and manage rare diseases. A rubric grades their diagnoses and plans. Expert 43.8% Not in cards
RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. Expert 94.2% Not in cards
Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. Expert 1,637 System cards only
Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. Expert 86% In use
Realm Financial Reasoning Agents analyze financial documents and spreadsheets. A model grades their answers against practitioner rubrics. Expert 56.2% Not in cards
RepoInject v1 Coding agents do benign tasks in 84 repository tasks seeded with prompt injections. Reports attack success rate at one and three attempts. Expert 23.8% System cards only
ReproBAIT Agents independently reproduce biological AI models from published scientific papers. Scored as success rate. Expert 85% System cards only
ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. Research frontier 64.7% System cards only
RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. Research frontier 0% System cards only
Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. Everyday Not in cards
SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. Expert 90.7% System cards only
SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. Expert 58.7% System cards only
SheetBench Agents solve finance and accounting spreadsheet tasks through a browser in a fresh Google Sheet. The score is the share of expected answer cells that match exactly. Expert 64.7% Not in cards
SimulatorArena Models act as students or writing clients. Researchers compare their messages and assistant ratings with those from people who used the same assistants. School 1.01 Not in cards
SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). Expert Not in cards
Speko LLM Benchmark for Voice Agents v3 Models use simulated support tools to look up orders and issue refunds. The task score gives partial credit when a model stops early without causing harm. Everyday 94.7% Not in cards
Speko Speech-to-Speech Benchmark (six concierge scenarios) Voice agents book flights, move reservations, and revise budgets during scripted calls. Speko checks whether agents wait for confirmation and use the latest details. Everyday 0.91 Not in cards
SPOT5 daily photograph scheduling Agents choose which photographs a satellite should take and assign cameras. Feasible schedules earn points for image priority while respecting instrument, data rate, and storage limits. University Not in cards
STEM Professor Bench Agents solve unpublished research problems written by professional scientists. Most problems concern theoretical physics, with some partial credit. Research frontier 61.8% System cards only
StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. Expert 0.50% In use
SuperCLUE-Agent Scenario Agents complete tasks drawn from Chinese application scenarios. University 96.6 Not in cards
SuperCLUE-DeepResearch Agents research questions and assemble reports in Chinese. University Not in cards
SuperCLUE-DeepSearch Agents search for information to answer Chinese research questions. University 74.3 Not in cards
SuperCLUE-DeepSWE Agents complete extended software engineering tasks in Chinese. Expert 84.3 Not in cards
SuperCLUE-LarkClaw Claw assistants complete tasks through Feishu. Everyday Not in cards
SuperCLUE-Office Agents process documents, analyze data, and write office materials. University Not in cards
SuperCLUE-OpenClaw Models complete coding, memory, and writing tasks in the OpenClaw environment. University 92.3 Not in cards
SuperCLUE-R1 Search DeepSeek R1 services answer questions that require search and reasoning. University Not in cards
SuperCLUE-SWE Agents complete software engineering tasks in Chinese. Expert 73.3 Not in cards
SuperCLUE-SWE with Claude Code Models solve Chinese software engineering tasks in the Claude Code harness. Expert 75.8 Not in cards
SuperCLUE-SWE with mini-swe-agent Models solve Chinese software engineering tasks with the mini-swe-agent harness. Expert 60.0 Not in cards
SuperCLUE-Terminal Agents complete tasks in a terminal. Expert 64.7 Not in cards
SuperCLUE-WeClaw Claw assistants complete tasks through WeChat. Everyday Not in cards
SuperCLUE-XClaw Claw assistant products complete user tasks in Chinese. Everyday Not in cards
SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. Expert System cards only
SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. Expert 0% System cards only
Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. Everyday 11.8% System cards only
TissueMicroscopyBench Agents examine tissue images and answer questions about cells, staining, and changes over time. Expert 34% Not in cards
TREC BioGen 2024 Systems retrieve biomedical literature and synthesize evidence into reports that answer a question. Expert 79.9 Not in cards
TxBench-Antibody-Development Agents use evidence to make manufacturing, quality, and clinical development decisions about candidate therapeutic antibodies. Expert 61.1% Not in cards
UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. System cards only
UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. Expert 15.6 System cards only
Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. Expert 21.1% System cards only
Uncritically reporting flawed results (Anthropic) Agents analyze data with code that silently mishandles broken values. Graders check whether the agent flags the flaw or reports the biased numbers. Expert 9.83 System cards only
vCX Benchmark Models run inside a Retell voice agent on pass or fail checks for business rules, grounding, tool calls and speech-ready replies. Each suite reports its own pass rate. Everyday 93.3% Not in cards
WebsiteBench Agents explore reference websites and rebuild their appearance, behavior, and persistent state. Hidden browser tests require functional correctness and screenshot similarity, then report the fraction passed. Expert 26.2% Not in cards
White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. Research frontier System cards only
xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. Not in cards