Robotics
Models control a real robot or a simulated one. We count 394 benchmarks here, 9 of them arenas. A benchmark belongs here when environment includes a physical robot or device. Or environment includes a simulator, and output includes actions.
How fast it grows
174 were released since July, more than the 86 in the first half of 2026.
How much is still open
No model has passed 85% on 166 of the 192 benchmarks here that score in percent, 86% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| 3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. | Expert | Nov 2026 | 59.5 | Not in cards |
| AbMaze Agents make choices in a simulated antibody optimization experiment under a fixed budget. Recorded assay results determine normalized improvement on 200 test episodes. | Expert | Nov 2026 | Not in cards | |
| ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. | Everyday | Nov 2026 | | Not in cards |
| ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. | Everyday | Nov 2026 | Not in cards | |
| AstroMemBench Spacecraft agents use information from earlier observations to complete simulated missions. The environment scores task completion, safety, and resource use. | University | Nov 2026 | Not in cards | |
| Crazyrouter AI Trading Arena Models buy, sell, or hold three cryptocurrencies every five minutes using simulated accounts. The grader values portfolios at live quotes and deducts trading fees. | University | Nov 2026 | -7.45% | Not in cards |
| CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. | Everyday | Nov 2026 | | Not in cards |
| DeGennaro Workplace sample Agents act as an employee’s assistant using 80 simulated workplace tools and anonymized company records. Four tasks grade the final state. | Expert | Nov 2026 | Not in cards | |
| DrivingBench v2 Vision-language models drive a real Tesla through two unseen cone courses with steering, accelerator and brake tools. The score is the furthest progress over three attempts in one conversation, averaged over six trials. | Everyday | Nov 2026 | | Not in cards |
| Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. | University | Nov 2026 | Not in cards | |
| EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. | University | Nov 2026 | Not in cards | |
| ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. | Everyday | Nov 2026 | Not in cards | |
| FrontierBreach Agents perform simulated enterprise tasks while processing untrusted material. The benchmark checks whether each agent resists instructions that conflict with its assigned task. | Expert | Nov 2026 | | Not in cards |
| FrontierEE Bench Agents run electrical engineering experiments on real lab hardware through networked instruments, camera photos and tickets to a human operator. Deterministic verifiers check 10 tasks against protected bench records. | Expert | Nov 2026 | 36.7% | Not in cards |
| GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. | Expert | Nov 2026 | 16.4% | Not in cards |
| GeoGuesser Bench Agents explore offline Street View panoramas to report coordinates, name places, and find objects. Coordinate guesses earn distance points. Other answers require an exact match. | Everyday | Nov 2026 | | Not in cards |
| GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. | Everyday | Nov 2026 | | Not in cards |
| GraspBench Robots propose grasps and try to lift objects in MuJoCo. Each attempt must pass motion planning and execution on the chosen robot. | Everyday | Nov 2026 | Not in cards | |
| HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. | Everyday | Nov 2026 | 18.8% | Not in cards |
| HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. | Everyday | Nov 2026 | 61.9% | Not in cards |
| HW-Bench Agents complete six hardware engineering tasks for a 2.45 GHz antenna using native design and simulation tools. Code checks and calibrated model judges grade outputs against engineer-written criteria. The headline pass rate measures runs that satisfy every critical criterion, averaged across tasks. | Expert | Nov 2026 | 67% | Not in cards |
| Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. | Expert | Nov 2026 | 83.9% | Not in cards |
| Malicious agentic influence campaigns, 34-point rubric (Anthropic) Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale. | Expert | Nov 2026 | System cards only | |
| ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. | Everyday | Nov 2026 | Not in cards | |
| MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. | Everyday | Nov 2026 | 50.4% | Not in cards |
| Parley Labs negotiation tasks Agents negotiate contracts and plan around market changes with scripted counterparties. An offline verifier grades outcomes. Ten public development tasks accompany private scored sets. | Expert | Nov 2026 | Not in cards | |
| Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. | Expert | Nov 2026 | | Not in cards |
| Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. | Expert | Nov 2026 | 0.60 | Not in cards |
| RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. | Everyday | Nov 2026 | | Not in cards |
| RoboCasa-Mem Robots complete nine household tasks that require memory. Later steps depend on objects the robot saw or actions it took earlier. | Everyday | Nov 2026 | 16% | Not in cards |
| RoboColosseum Physical robots execute fresh instructions in paired blind trials. Human evaluators choose the better run or a tie, and comparisons determine ratings for each embodiment. | Everyday | Nov 2026 | Not in cards | |
| RoboDx Agents repair faults in robot navigation software. Hidden driving scenarios test the repair. A separate diagnosis track asks models to identify faults without seeing the code. | Expert | Nov 2026 | 100% | Not in cards |
| RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. | University | Nov 2026 | 80.6% | Not in cards |
| Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. | Everyday | Nov 2026 | 42.9% | Not in cards |
| Sequitur Research Lab Automation Agents run simulated wet-lab automation protocols. Scored out of 100 on how well each experimental phase is carried out, averaged over six runs. | Research frontier | Nov 2026 | 74.5 | System cards only |
| ShadowBench Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match. | Everyday | Nov 2026 | 48.8 | Not in cards |
| SMB Bench An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule. | University | Nov 2026 | $7,749 | Not in cards |
| UAV Robust Control Hub Agents construct policies for a fictional civilian drone simulator. The benchmark evaluates policies on hidden scenarios and reports separate results under changed conditions. | University | Nov 2026 | Not in cards | |
| VABAR Agents program robots to sort tools, mesh gears, and handle cables in simulation. The harness checks success as object shapes and positions vary. | University | Nov 2026 | Not in cards | |
| WeChat Pay Physical Client Evaluation Agents operate the WeChat Pay client on physical phones. The team scores execution paths, use of supplied knowledge, acceptance checks, and account creation accuracy. | University | Nov 2026 | Not in cards | |
| AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. | Everyday | Oct 2026 | Not in cards | |
| AdaExploreBench Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice. | Everyday | Oct 2026 | | Not in cards |
| AML Bench (Veris AI) Agents investigate suspicious activity at a simulated bank. A model judge checks whether the recorded service requests and final case disposition satisfy every requirement. | Expert | Oct 2026 | 70.8% | Not in cards |
| Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. | Expert | Oct 2026 | Not in cards | |
| Argo-Bench Agents query a simulated food delivery company's warehouse and act on their findings. The grader measures the business consequences of their decisions across 210 tasks. | Expert | Oct 2026 | 70.2 | Not in cards |
| AutoVend Agents buy stock and schedule deliveries for 20 simulated vending machines over 180 days. They share one warehouse and pay for each site visit. | University | Oct 2026 | 309.4 | Not in cards |
| BARC-Bench Agents handle routine workplace requests and use records to infer privacy and safety constraints. A judge checks whether each attempt completes safely or defers. | University | Oct 2026 | 72.7% | Not in cards |
| BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. | Everyday | Oct 2026 | 0.56 | Not in cards |
| Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. | Expert | Oct 2026 | 37.8% | In use |
| CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. | Expert | Oct 2026 | Not in cards | |
| Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. | Expert | Oct 2026 | $9,514 | Not in cards |
| ClothDojo Robot policies use both hands to flatten and fold garments. The simulator checks success on familiar garments and garments withheld from training. | Everyday | Oct 2026 | 44.7% | Not in cards |
| CRM-Agent-Bench Agents handle customer and business requests in a simulated enterprise. Executable checks verify tool arguments, action order, and which records changed across 555 tasks. | University | Oct 2026 | | Not in cards |
| CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. | Expert | Oct 2026 | 87.7% | Not in cards |
| DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. | Everyday | Oct 2026 | 40% | Not in cards |
| DexArena Robots use both hands to complete household and industrial tasks. The simulator checks each stage and the final outcome across ten multistage tasks. | Everyday | Oct 2026 | 52.1% | Not in cards |
| Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. | Everyday | Oct 2026 | Not in cards | |
| Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. | Expert | Oct 2026 | 59.8 | Not in cards |
| Embodied Agent Arena Vision-language agents solve geometry, spatial reasoning, affordance, planning, and manipulation tasks. The 1,000-case evaluation combines 32 established sources with GeoProbe. | Everyday | Oct 2026 | Not in cards | |
| EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. | Everyday | Oct 2026 | 52% | Not in cards |
| EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. | University | Oct 2026 | 41.4% | Not in cards |
| FlashDexRetarget 50-motion benchmark Retarget 50 human hand-object motions (HOT3D, TACO, OakInk2) to XHand and Sharpa hands in Isaac Sim. Scored by object pose-error success thresholds. | Everyday | Oct 2026 | Not in cards | |
| Hedge Gym Models adjust an options hedge each day over a 21-day price path. The evaluator measures replication error and transaction costs on synthetic or historical markets. | Expert | Oct 2026 | Not in cards | |
| HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. | Everyday | Oct 2026 | | Not in cards |
| Hy-Nomas-Bench Interactive video world models follow text and camera controls across multiple turns and retain spatial memory. | Expert | Oct 2026 | Not in cards | |
| InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. | Everyday | Oct 2026 | 41.7% | Not in cards |
| LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. | Everyday | Oct 2026 | 63.3% | Not in cards |
| Learn2Play Bench Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure. | Everyday | Oct 2026 | 81.8 | Not in cards |
| LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. | Expert | Oct 2026 | 66.5% | Not in cards |
| MAMUBench Agents represent different users who share a limited resource. The benchmark compares their combined outcome with one agent coordinating for everyone. | Everyday | Oct 2026 | Not in cards | |
| Navital Gym Agents complete tasks in six simulated Indian service settings. This port covers UPI Desk, which grades 50 payment and dispute tasks using written rules. | Everyday | Oct 2026 | 100% | Not in cards |
| OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. | Everyday | Oct 2026 | Not in cards | |
| PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. | Research frontier | Oct 2026 | 14% | Not in cards |
| Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. | Everyday | Oct 2026 | | Not in cards |
| PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. | Everyday | Oct 2026 | | Not in cards |
| Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. | Expert | Oct 2026 | 32.7% | Not in cards |
| Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. | Everyday | Oct 2026 | Not in cards | |
| RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. | Everyday | Oct 2026 | 19.1% | Not in cards |
| RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. | Expert | Oct 2026 | 0.06 | Not in cards |
| RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. | Everyday | Oct 2026 | 17% | Not in cards |
| RWE-bench Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement. | Expert | Oct 2026 | 62% | Not in cards |
| Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. | Expert | Oct 2026 | $2,949 | Not in cards |
| SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. | Expert | Oct 2026 | 100% | Not in cards |
| SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. | Expert | Oct 2026 | Not in cards | |
| SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. | Expert | Oct 2026 | 25.1% | Not in cards |
| SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. | Expert | Oct 2026 | | Not in cards |
| SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. | Expert | Oct 2026 | | Not in cards |
| SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. | Expert | Oct 2026 | 30.5% | Not in cards |
| Solana Machine Trading Agents write trading bots for hidden market scenarios replayed on offline Solana snapshots. The headline score compares total profit and loss against a reference bot. | Expert | Oct 2026 | 6.11 | Not in cards |
| StoreBench Agents manage a simulated apparel store in Medusa. They order inventory, change prices, ship purchases, and process refunds as demand changes. | University | Oct 2026 | 0.83 | Not in cards |
| TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. | Everyday | Oct 2026 | Not in cards | |
| TavernBench Agents build a Hearthstone Battlegrounds player through research, simulator updates, and live games. | Expert | Oct 2026 | 95.6 | Not in cards |
| The Impossible Refund Assistants pursue a late train refund in a simulated email environment. Separate judges rate deception and other behaviors from the messages. | Everyday | Oct 2026 | Not in cards | |
| Unsatz Agents design experiments and propose equations for an unknown simulated system. Their final program must predict held-out interventions and conditions beyond the original experiments. | Expert | Oct 2026 | 60% | Not in cards |
| Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. | Expert | Sep 2026 | 0.17% | System cards only |
| ArtifactArena Models design robot bodies, controllers and strategy for a simulated sumo-style arena. Artifacts fight head-to-head and models get Elo ratings. | Expert | Sep 2026 | 1,539 | Not in cards |
| CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. | Expert | Sep 2026 | | Not in cards |
| Cloud Expansion CE80-10 Agents complete a cloud network security exercise. The verifier checks the resulting cloud configuration and observed network behavior. | Expert | Sep 2026 | Not in cards | |
| Cloud Expansion RX00 Agents complete a simulated cloud reconnaissance assessment. The verifier compares the submitted result with the recorded state of the exercise. | Expert | Sep 2026 | Not in cards | |
| CorpBench Work Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly. | Everyday | Sep 2026 | 92.7 | Not in cards |
| DolphinBench Agents use long-term memory to complete 600 tool-using tasks for three simulated knowledge workers with roughly 500,000 user-message tokens each. | Expert | Sep 2026 | 70.7% | Not in cards |
| DrivingBench Vision-language models drive a real Toyota Corolla through a cone course with up to three attempts in one conversation. Scored on the furthest progress along the course. | Everyday | Sep 2026 | 100% | Not in cards |
| DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. | Expert | Sep 2026 | 40.0 | Not in cards |
| EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. | Research frontier | Sep 2026 | 29.8 | Not in cards |
| ExplorationBench Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help. | Research frontier | Sep 2026 | 89% | Not in cards |
| GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. | Everyday | Sep 2026 | 80.2 | Not in cards |
| GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Scored by a distance-based GeoGuessr-style reward from 0 to 1. | Everyday | Sep 2026 | 0.64 | Not in cards |
| GlyphBench Agents choose actions from text-rendered game boards. The standard suite contains 303 tasks across six game collections and scores agents by cumulative return. | University | Sep 2026 | | Not in cards |
| HarvestBench Agents drive simulated tractors and decide how much fuel to spend avoiding animals. The benchmark measures how often they hit animals as avoidance costs change. | Everyday | Sep 2026 | 0.40% | Not in cards |
| Huayu Awards AI Cockpit Evaluation 2026 In-car assistants interpret requests and operate vehicle features. Tests include planning three-stop trips and adjusting several cabin controls from one request. | Everyday | Sep 2026 | Not in cards | |
| ironbench Agents write firmware for tasks such as serial communication and controller sequencing. The grader checks device behavior and counts tasks solved within the allowed attempts. | University | Sep 2026 | 69.2% | Not in cards |
| KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. | Expert | Sep 2026 | 29.4% | Not in cards |
| Long-Horizon Medical Agent Benchmark, Veterinary Clinic Ops v2 Agents operate a simulated veterinary clinic through its database API. Each of the 98 tasks checks whether the agent leaves the clinic records in the required state. | Expert | Sep 2026 | 100% | Not in cards |
| MANDOS Session 2 Models make process safety decisions across four private industrial scenarios. A model judge assesses each run. The leaderboard reports the share of passing runs. | Expert | Sep 2026 | 96.9% | Not in cards |
| McStasBench Agents design neutron scattering instruments. A simulator checks whether their designs reproduce published measurements. The authors score two additional instruments separately. | Expert | Sep 2026 | 43.8% | Not in cards |
| metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. | Everyday | Sep 2026 | Not in cards | |
| Nego Eval Agents choose negotiation partners and remember prior encounters across games. The evaluator compares match scores with reference policies on the same seeds. | Everyday | Sep 2026 | Not in cards | |
| OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. | Everyday | Sep 2026 | | Not in cards |
| OpenRSI Index Isaac Lab PegInsert Reward Search Agents design robot training rewards under a fixed budget. The score is held-out peg-insertion success as a fraction. | Research frontier | Sep 2026 | 0.60 | Not in cards |
| Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. | Expert | Sep 2026 | System cards only | |
| PACT (robot arm orchestration) Agents write and revise programs to control a simulated robot arm. The simulator checks whether the arm completes the requested manipulation. | Expert | Sep 2026 | Not in cards | |
| Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. | Everyday | Sep 2026 | 36.3% | Not in cards |
| PhoneShell iOS Agent Benchmark Agents operate apps on a physical iPhone to set timers, find books, and complete other everyday tasks. Automated checks read the resulting device state. | Everyday | Sep 2026 | 96.6% | Not in cards |
| Prosus Vending Bench v2 Agents run six simulated vending machines across three locations for 30 days. They order stock, set prices, and collect cash. The score measures final bank balance in euros. | University | Sep 2026 | 4497.2 | Not in cards |
| Qwen-Audio-Agent Smart Cockpit Benchmark Agents operate simulated vehicle, navigation, music, and weather services. The short suite checks whole cases. The long suite checks tool use on each turn. | Everyday | Sep 2026 | | Not in cards |
| ReactHuman Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate. | Everyday | Sep 2026 | 63.7% | Not in cards |
| REVERSAL-BENCH Reset-free reinforcement learning agents learn manipulation policies across environments with controlled reversibility and recoverability checks. | Expert | Sep 2026 | Not in cards | |
| Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. | Everyday | Sep 2026 | 52.5% | Not in cards |
| RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. | Everyday | Sep 2026 | 65.7% | Not in cards |
| Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. | Everyday | Sep 2026 | 20% | System cards only |
| SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. | Expert | Sep 2026 | 45.3% | Not in cards |
| SemiKongBench-100 Agents resolve 100 synthetic semiconductor operating exceptions. The evaluator grades each workflow with deterministic checks. | Expert | Sep 2026 | Not in cards | |
| StationeryBench Robots use both arms to handle common desk supplies. A human grades progress from zero to 100 on five tasks. | Everyday | Sep 2026 | 46.0 | Not in cards |
| UK AISI Out of Scope Supply Chain Attack Agents in simulated hard cyber challenges with incidental internet access may attack out-of-scope open source projects. Scored as rate of samples with such attacks. | Expert | Sep 2026 | 12% | System cards only |
| VAmoS Pro Bench Voice agents handle several utility billing requests in one call. A model checks the spoken amounts and recorded actions against the requirements for each request. | University | Sep 2026 | | Not in cards |
| WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. | Everyday | Sep 2026 | 42.3% | Not in cards |
| ASI-Bench AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge. | Research frontier | Aug 2026 | 55.8 | Not in cards |
| Blobfish Litigation Bench Agents complete 100 tasks in a simulated litigation practice. The grader compares the changes agents make to case records against required outcomes. | Expert | Aug 2026 | Not in cards | |
| CounselBench-100 Agents investigate legal matters and decide which actions the evidence supports. The verifier checks decisions, changes to case records, and advice across 100 synthetic tasks. | Expert | Aug 2026 | Not in cards | |
| DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. | Everyday | Aug 2026 | 71.4% | Not in cards |
| E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. | University | Aug 2026 | 14.3 | Not in cards |
| EduClaw-Bench Agents tutor a simulated learner over 30 days. The benchmark tracks learning gains, responsiveness, helpfulness, and curriculum design across 55 scenarios. | University | Aug 2026 | Not in cards | |
| EmbodiedCLUE-VLA RoboTwin Evaluation Robot policies perform ten two-handed manipulation tasks under randomized RoboTwin 2.0 Hard conditions. The score averages task success rates over twenty episodes per task. | Everyday | Aug 2026 | Not in cards | |
| FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. | Expert | Aug 2026 | 20.6% | Not in cards |
| HEPToolBench 1.2 Models generate inputs for particle physics simulations and inspect the resulting files. The main suite has 28 tasks and a separate extension adds three debugging tasks. | Expert | Aug 2026 | | Not in cards |
| HM3D-MFMON Agents navigate across floors and find a sequence of objects. | Everyday | Aug 2026 | Not in cards | |
| HWE-bench CAE Agents run engineering simulations, extract numerical results, and report them. A verifier repeats the extraction from the solver output and checks engineering tolerances. | Expert | Aug 2026 | 0.90% | Not in cards |
| ITSMBench Agents operate an IT service desk across 53 conversations with simulated colleagues. The verifier grades the resulting database state, including permissions and required actions. | University | Aug 2026 | 0.70 | Not in cards |
| LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. | Everyday | Aug 2026 | | Not in cards |
| MIVAS Bench Voice agents handle calls for a dermatology practice, a law firm, and a retailer. Each call must pass checks on tools, specialist handoffs, and database changes. | University | Aug 2026 | 56.5% | Not in cards |
| MobilePA-Bench Agents complete 1,705 mobile planning tasks using tools, persistent memory, reusable skills, and specialist agents. Task-specific checks evaluate tool calls, state changes, and interaction behavior. Additional gates check required memory access and skill loading. | Everyday | Aug 2026 | 76.9% | Not in cards |
| S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. | Expert | Aug 2026 | Not in cards | |
| Agentic Trader Agents search for market information and place simulated foreign exchange orders using live prices. Each agent manages its own positions, cash, and margin. | University | Jul 2026 | | Not in cards |
| ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. | Everyday | Jul 2026 | 47.6% | Not in cards |
| BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. | Everyday | Jul 2026 | Not in cards | |
| Bot Analysis Arena Models write trading rules for simulated US stock accounts. Some agents revise their rules after trades. Software tests each revision against the current rules before adoption. | University | Jul 2026 | Not in cards | |
| DexVerse Robot policies perform tabletop manipulation tasks in Isaac Lab, including tool use, bimanual coordination, and sequences with several goals. Task-specific success conditions check completion under different visual settings and robot embodiments. | Everyday | Jul 2026 | 34% | Not in cards |
| Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. | Expert | Jul 2026 | 97.3% | Not in cards |
| EvoPolicyGym Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves. | Expert | Jul 2026 | 0.89 | Not in cards |
| Feishu Office v1 Agents handle 140 Chinese workplace requests in a simulated Feishu workspace. The grader checks the resulting calendars, documents, tables, and approvals. | University | Jul 2026 | Not in cards | |
| Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. | Everyday | Jul 2026 | Not in cards | |
| Game Arena Bargaining Two language models negotiate how to split a pool of items, each with private item values. Each side tries to reach a good deal without giving away its values. | Everyday | Jul 2026 | | Not in cards |
| GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. | Everyday | Jul 2026 | Not in cards | |
| Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. | Everyday | Jul 2026 | Not in cards | |
| HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. | Everyday | Jul 2026 | | Not in cards |
| InfraBench Agents deploy, profile, debug, and optimize model workloads on AMD MI35X GPUs. Task-specific verifiers grade 20 engineering assignments. | Expert | Jul 2026 | Not in cards | |
| MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. | Expert | Jul 2026 | | Not in cards |
| OptimBench Dispatch Agents route vehicles under capacity and time constraints, then revise their routes after a disruption. | University | Jul 2026 | 0.92 | Not in cards |
| RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. | Everyday | Jul 2026 | 27.6% | In use |
| SpecDecode RL Agents choose how many tokens to draft during speculative decoding. An offline grader measures speedup over the target model while requiring identical output. | Expert | Jul 2026 | Not in cards | |
| Speko Speech-to-Speech Benchmark (20-turn concierge call) Voice agents try to book a flight, handle a failed booking, and remember a changed date. Eight automated checks score their actions and replies. | Everyday | Jul 2026 | 1.00 | Not in cards |
| Task AW IT support Agents complete five support workflows in simulated IT systems. Task checks cover onboarding, offboarding, and cleanup of accounts and assets. | University | Jul 2026 | Not in cards | |
| UAV Operator Models manage simulated drone missions as weather and flight restrictions change. The simulator scores their decisions from logged aircraft and mission state. | Expert | Jul 2026 | Not in cards | |
| VAmoS Bench Voice agents answer simulated bank customers and use tools to manage credit cards. A model checks each call against its transcript and recorded actions. | University | Jul 2026 | | Not in cards |
| Act2Answer Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control. | Everyday | Jun 2026 | | Not in cards |
| AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. | Everyday | Jun 2026 | 67.8% | Not in cards |
| ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. | Everyday | Jun 2026 | 48.2% | Not in cards |
| CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. | Expert | Jun 2026 | $22,148,357 | Not in cards |
| CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. | University | Jun 2026 | Not in cards | |
| Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. | Expert | Jun 2026 | 0.20 | Not in cards |
| DuetBench Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head. | Expert | Jun 2026 | 1,035 | Not in cards |
| DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. | Everyday | Jun 2026 | 10.6% | Not in cards |
| EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. | Everyday | Jun 2026 | Not in cards | |
| FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. | Research frontier | Jun 2026 | 38.1% | In use |
| iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. | Everyday | Jun 2026 | 51.9% | Not in cards |
| LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. | Expert | Jun 2026 | 0.66 | Not in cards |
| MANDOS Run 003 Models make decisions during seven industrial incidents and four fictional scenarios. A model judge checks their decisions, including whether they seek expert advice. | Expert | Jun 2026 | 100% | Not in cards |
| MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. | Expert | Jun 2026 | 69.2 | Not in cards |
| MetroLLM-Bench Agents operate a transit kiosk using tools for routes, fares, and service disruptions. The first score checks their actions automatically. The composite also includes model judgments. | Everyday | Jun 2026 | | Not in cards |
| MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. | Everyday | Jun 2026 | Not in cards | |
| OmniGameArena Models play games while response latency advances the game clock. | Everyday | Jun 2026 | 0.47 | Not in cards |
| PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. | Expert | Jun 2026 | | Not in cards |
| RadioBench Models answer radio engineering questions and generate programs for software-defined radios. The authors measure successful execution, hardware deployment, and reception of the intended signal. | University | Jun 2026 | 97% | Not in cards |
| Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. | Expert | Jun 2026 | 62.1% | Not in cards |
| RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. | Everyday | Jun 2026 | | Not in cards |
| Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. | Everyday | Jun 2026 | Not in cards | |
| SentinelBench Agents monitor changing web environments and respond when scripted events make progress possible. Scores measure completion, reaction time and resource use. | Expert | Jun 2026 | Not in cards | |
| SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. | Everyday | Jun 2026 | Not in cards | |
| SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. | Everyday | Jun 2026 | 14.4% | Not in cards |
| SpatialWorld Digital Agents complete entertainment tasks in digital environments. The score is the published digital task success rate. | Everyday | Jun 2026 | | Not in cards |
| SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. | Everyday | Jun 2026 | 14.4% | Not in cards |
| STATE-Bench Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. | Expert | Jun 2026 | 58.9% | Not in cards |
| UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. | Everyday | Jun 2026 | 55.8% | Not in cards |
| AerialSim Agents fly a simulated drone using camera observations or write a program for the flight. The simulator checks the recorded trajectory against the mission requirements. | Expert | May 2026 | 73.2% | Not in cards |
| AgentEscapeBench Agents solve procedural escape rooms using unfamiliar tools. The evaluator checks whether they complete dependent steps and escape. | Everyday | May 2026 | 76.9% | Not in cards |
| Agentick Agents solve procedurally generated gridworld tasks involving navigation, planning, reasoning, memory, generalization, and coordination. Scores normalize task performance between random and oracle reference policies. | Everyday | May 2026 | 0.63 | Not in cards |
| ASPI Agents complete simulated tasks with incomplete instructions. The authors compare adherence to user intent across matched versions of each task. | Everyday | May 2026 | | Not in cards |
| BoardGameBench (Homer Quan) Models play seven board game variants against a search engine. The scorer rewards quick wins and gives partial credit for longer losses. | Everyday | May 2026 | | Not in cards |
| CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. | Expert | May 2026 | 54.7% | Not in cards |
| ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. | Everyday | May 2026 | 70% | Not in cards |
| HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. | Everyday | May 2026 | Not in cards | |
| NARRA-Gym Agents build interactive stories around a user’s experiences and adapt scenes as the conversation unfolds. Judges rate story quality and user experience separately. | Everyday | May 2026 | 2 | Not in cards |
| PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. | Expert | May 2026 | 50% | System cards only |
| RoboChallenge Table30 v2 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. | Everyday | May 2026 | 0.73% | Not in cards |
| RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. | Everyday | May 2026 | 21.5% | Not in cards |
| Science Sim Chemistry Models compare simulated compounds and choose whether to advance or reject them. The evaluator scores decisions against measured reaction outcomes and a budget. | University | May 2026 | Not in cards | |
| TERMS-Bench Agents negotiate the price of a single item with simulated counterparts over up to 10 rounds of alternating offers. Counterparts follow six behavior families across three regimes. A program scores each agent from its logged actions on four separate axes: surplus captured, agreement calibration, belief error about the opponent and procedural violations. Extra modes score merchant profit and a bankroll over many sessions. | Everyday | May 2026 | 0.69 | Not in cards |
| TVRBench Agents move through simulated indoor scenes to reproduce a target image's viewpoint. Success requires stopping at exactly the target pose on the simulator's discrete grid. The evaluation split has 500 tasks across 80 scenes. | Everyday | May 2026 | | Not in cards |
| WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. | May 2026 | | Not in cards | |
| Agent Bazaar Agents price goods or trade with sellers in simulated markets. The benchmark measures whether their decisions sustain prices, deter fraud, and preserve welfare and profits. | University | Apr 2026 | Not in cards | |
| AutomationBench Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. | University | Apr 2026 | | In use |
| Clash of AIs Season 1 Models publish trade entries, targets, and stops before prices move. Clash of AIs simulates fills against market prices and ranks accounts by total equity. | University | Apr 2026 | $283,903 | Not in cards |
| HWE-Bench v1.0 (RTL bug repair) Agents fix bugs in six open-source hardware projects. The benchmark reports the share of fixes that pass repository tests in simulation. | Expert | Apr 2026 | 79.9% | Not in cards |
| KellyBench Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth. | Expert | Apr 2026 | -7.90% | Not in cards |
| RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. | Everyday | Apr 2026 | 46.2% | In use |
| Spatial-Gym Models find paths that satisfy grid puzzle rules. The evaluation uses 500 SPaRC puzzles in one-shot and stepwise settings, including a setting that allows backtracking. | School | Apr 2026 | Not in cards | |
| UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. | Research frontier | Apr 2026 | 80% | System cards only |
| VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. | Everyday | Apr 2026 | | Not in cards |
| Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. | Expert | Apr 2026 | System cards only | |
| ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. | Everyday | Mar 2026 | 100% | In use |
| AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. | Expert | Mar 2026 | 76.5% | Not in cards |
| BarterBench Agents trade scarce goods to meet inventory targets. The hub port assigns one model-controlled trader to a marketplace of scripted opponents. | University | Mar 2026 | Not in cards | |
| CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. | University | Mar 2026 | | Not in cards |
| GUI-CEval Models answer questions about Chinese app screens, locate controls, and choose actions. Agents also complete tasks on physical devices, where the grader checks interface elements. | Everyday | Mar 2026 | | Not in cards |
| LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. | Everyday | Mar 2026 | 71.4% | Not in cards |
| ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. | Everyday | Mar 2026 | Not in cards | |
| PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. | Everyday | Mar 2026 | 75.1% | Not in cards |
| RetailBench Agents operate a simulated supermarket for up to 180 days. They choose suppliers, order stock, and set prices as goods age and customers respond. | University | Mar 2026 | $24,351 | Not in cards |
| RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. | Everyday | Mar 2026 | 10.4% | Not in cards |
| RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. | Everyday | Mar 2026 | | Not in cards |
| RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. | Everyday | Mar 2026 | | Not in cards |
| AI GameStore Vision-language models play short episodes of 100 games adapted from Apple App Store and Steam top charts. Each game score is divided by the average human score, and models are ranked by the geometric mean. | Everyday | Feb 2026 | | Not in cards |
| Blicket Test Causal Reasoning Models experiment with a simulated detector to infer which objects activate it. The hub version rewards correct identification and informative experiments. | Everyday | Feb 2026 | | Not in cards |
| CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. | Everyday | Feb 2026 | Not in cards | |
| EnterpriseBench: CoreCraft Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy. | Expert | Feb 2026 | 77.4% | In use |
| EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. | Expert | Feb 2026 | 51% | Not in cards |
| MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. | Everyday | Feb 2026 | | Fading |
| OdysseyArena Agents explore interactive environments built from four transition primitives and must work out the hidden dynamics over long horizons. OdysseyArena-Lite has 120 standard tasks. | Expert | Feb 2026 | | Not in cards |
| OdysseyArena AI Trading Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials. | Expert | Feb 2026 | 23.3% | Not in cards |
| OdysseyArena Energy Dispatch Agents manage energy dispatch under hidden rules. The score is average task success over four trials. | Expert | Feb 2026 | 23.3% | Not in cards |
| OdysseyArena Turn on Lights Agents learn hidden switches and turn on lights. The score is average task success over four trials. | Expert | Feb 2026 | 28.3% | Not in cards |
| SalesBench (Insurance Sales) Agents find insurance leads, quote plans, and schedule callbacks in a simulated sales pipeline. The evaluator rewards converted monthly premiums and checks compliance with calling restrictions. | University | Feb 2026 | Not in cards | |
| SimulCost Agents tune physics simulations to meet an accuracy target. The authors measure success and computational cost for an initial guess and for repeated attempts with feedback. | Expert | Feb 2026 | 80.8% | Not in cards |
| YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. | University | Feb 2026 | $3,445,495 | In use |
| CAR-bench Agents act as an in-car voice assistant that talks with an LLM-simulated driver and uses 58 connected tools for navigation, productivity, charging and vehicle control under set policies. Besides ordinary tasks, hallucination tasks remove a needed tool or fact, and disambiguation tasks need a clarifying question or a lookup. Scores reward consistent success across repeated runs. | Everyday | Jan 2026 | 70% | Not in cards |
| MK-Bench Agents schedule jobs when machines break down or orders arrive or disappear. The grader compares each completion time with the best known schedule. | University | Jan 2026 | | Not in cards |
| RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. | Everyday | Jan 2026 | 76.3% | Not in cards |
| ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. | Expert | Jan 2026 | | Fading |
| Strategy Bench Models compete in six social deduction and strategy game environments. | Everyday | Jan 2026 | Not in cards | |
| TradeRank Arena Models choose positions in cryptocurrencies and US stocks using shared market data and simulated capital. The leaderboard ranks portfolios by return. | University | Jan 2026 | 0% | Not in cards |
| CubeBench Agents solve Rubik's Cube tasks across symbolic, visual and partially observed settings to measure spatial reasoning and long-horizon state tracking. | Expert | Dec 2025 | Not in cards | |
| CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. | Expert | Dec 2025 | | In use |
| REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. | Everyday | Dec 2025 | 53.1% | Not in cards |
| SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. | Everyday | Dec 2025 | 57.8% | Not in cards |
| Solubility Optimization (anindya) Models choose chemical transformations to improve predicted solubility. The package includes a benchmark runner that measures improvement on evaluation examples. | University | Dec 2025 | Not in cards | |
| SpatialTree Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level. | Everyday | Dec 2025 | | System cards only |
| Tau2 Bench Verified Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. | University | Dec 2025 | Not in cards | |
| tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. | Everyday | Dec 2025 | 82.7% | Fading |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Expert | Nov 2025 | | System cards only |
| OASIS (agent safety) Agents handle multi-turn requests with different levels of complexity. Evaluators measure the proportion of safe responses. | University | Nov 2025 | 95.2% | Not in cards |
| Real-Time Reasoning Gym Models act in games while token generation consumes part of the game clock. | Everyday | Nov 2025 | | Not in cards |
| Tau2 Bench Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. | University | Nov 2025 | Not in cards | |
| Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. | Everyday | Nov 2025 | | In use |
| Vending-Bench Arena Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance. | Expert | Nov 2025 | | Not in cards |
| AgentBench FC Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format. | Everyday | Oct 2025 | 58.9% | Not in cards |
| BuildArena Agents design machines in the Besiege simulator from written requirements. The simulator tests whether a vehicle reaches its target, how much weight a bridge supports, and how a lifting machine performs. | University | Oct 2025 | 45.3% | Not in cards |
| BuilderBench Agents learn to control a simulated robot that builds target block structures through physical interaction and exploration. | Expert | Oct 2025 | 98% | Not in cards |
| Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. | Everyday | Oct 2025 | 37% | Not in cards |
| LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. | Everyday | Oct 2025 | | Not in cards |
| LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. | Everyday | Oct 2025 | 53% | Not in cards |
| Magentic Marketplace Agents search for sellers, exchange proposals and make purchases. The simulation measures how much value customers receive after paying for their purchases. | Everyday | Oct 2025 | Not in cards | |
| RoboChallenge Fine-tuned policies run 10 real rollouts on each of 30 tabletop tasks across four robot types. Scored by success rate and progress. | Everyday | Oct 2025 | 42.7% | Not in cards |
| StockBench Agents trade stocks over a fixed evaluation window. StockBench ranks models by the mean of final return, maximum drawdown and Sortino ratio. The score is the board's rank, so a lower score is better. | Expert | Oct 2025 | 1.00 | Not in cards |
| USIM underwater robot tasks Agents follow instructions to navigate, grasp, transport, and track objects underwater. The simulator measures task completion across 700 trials. | Everyday | Oct 2025 | 37.6% | Not in cards |
| World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. | Everyday | Oct 2025 | 64.8% | Not in cards |
| Automotive-ENV Agents operate climate controls in an Android Automotive OS simulator. A validator checks the screen and vehicle signals after requests such as turning on the defroster. | Everyday | Sep 2025 | 49.3% | Not in cards |
| BEHAVIOR Challenge 2025 50 long-horizon household activities on a Galaxea R1 Pro in OmniGibson, trained from 10,000 demos. Scored by Q-score and success. | Everyday | Sep 2025 | Fading | |
| MAPP (Multi-Agent Path Planning) Models move several agents across a grid without collisions. The environment tracks how many agents reach their goals and how closely their routes follow shortest paths. | School | Sep 2025 | 70.8% | Not in cards |
| Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. | Everyday | Sep 2025 | 42.1% | Fading |
| OceanGym Agents interpret underwater images and sonar, then navigate simulated search and inspection tasks. The simulator scores how closely they reach target locations. | University | Sep 2025 | 14.6 | Not in cards |
| VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. | Everyday | Sep 2025 | | Not in cards |
| DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. | Everyday | Aug 2025 | 39% | Not in cards |
| GEN-Bench Agents select job operations as conditions change in a simulated factory. The benchmark compares schedule completion times with the best known result for each instance. | University | Aug 2025 | | Not in cards |
| LiveTradeBench Agents allocate simulated portfolios across US securities and Polymarket contracts using streamed prices and news. The researchers track portfolio returns over successive trading days. | University | Aug 2025 | 12.1% | Not in cards |
| Asper-Bench Agents turn complex user requests into programs that execute multi-step actions in a simulated digital assistant environment. Automated checks compare the resulting state with the requested goal. | Expert | Jul 2025 | Not in cards | |
| Emergent Price-Fixing by LLM Auction Agents Models trade in simulated auctions and communicate with other participants. A judge examines game logs for evidence of price fixing. | Everyday | Jul 2025 | 19% | Not in cards |
| Group Binary Search (Riedl) Agents each choose an integer without seeing the other guesses. They adjust after hearing whether the group sum exceeds or falls below a hidden target. | Everyday | Jul 2025 | | Not in cards |
| LLM Economist Models set tax rates or choose working hours in a simulated economy. Researchers compare the social welfare achieved under different tax policies. | University | Jul 2025 | Not in cards | |
| OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. | Everyday | Jul 2025 | 53.4% | Not in cards |
| RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. | Everyday | Jul 2025 | | Fading |
| TextQuests Agents play 25 classic interactive fiction games. The score is harmful actions per 100 steps, where lower is better. The board also reports game progress. | Everyday | Jul 2025 | | System cards only |
| Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. | Expert | Jun 2025 | 0% | In use |
| CVDP Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs. | Expert | Jun 2025 | 64.1% | Not in cards |
| MCP-SIM Agents turn physics problems into FEniCS simulations. The tasks include heat flow, elastic deformation, and fracture. The authors count solved tasks. | University | Jun 2025 | Not in cards | |
| RoboArena Double-blind pairwise A/B trials of generalist DROID policies on real Franka arms, free-form tasks by volunteer evaluators. Elo-style rating. | Everyday | Jun 2025 | 1,608 | Not in cards |
| RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. | Everyday | Jun 2025 | 58.3% | Not in cards |
| tau2-bench Dual-control customer-service simulations (retail, airline, telecom) where agent and simulated user both act. Graded on final state. | Everyday | Jun 2025 | 98.9% | In use |
| V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. | Everyday | Jun 2025 | Not in cards | |
| Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. | Everyday | Jun 2025 | 84.9% | Not in cards |
| WorldPrediction Agents predict and plan in procedural environments with long horizons. Tasks test whether models maintain a useful world model while acting. | University | Jun 2025 | Not in cards | |
| FoamBench Agents create OpenFOAM cases from written specifications. The grader checks whether the solver finishes and whether the final flow field matches a reference simulation. | University | May 2025 | 22.5% | Not in cards |
| gg-bench Models play 126 games invented by a language model and receive game-specific outcomes. | Everyday | May 2025 | 36.3% | Not in cards |
| KORGym Models play 51 text and visual games designed to test interactive reasoning. | Everyday | May 2025 | 0.82 | Not in cards |
| LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. | Everyday | May 2025 | 72.4% | Fading |
| Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. | Expert | May 2025 | 0.63% | System cards only |
| VideoGameBench Vision-language models play 10 popular 1990s Game Boy and MS-DOS games in real time from raw frames. Scored by the share of each game completed. | Expert | May 2025 | | Not in cards |
| AI Engineering Lab simulation self-validation Agents predict how a mechanical system should behave, inspect simulation outputs, and decide whether the model is correct. The authors score these decisions with F1. | University | Apr 2025 | 0.72 | Not in cards |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | Expert | Apr 2025 | 93.3% | System cards only |
| m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. | Everyday | Apr 2025 | | Dropped |
| Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. | Expert | Apr 2025 | System cards only | |
| AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. | Everyday | Mar 2025 | | Not in cards |
| Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. | Everyday | Mar 2025 | Not in cards | |
| IVISTA Voice Interaction, revised 2023 protocol In-car assistants carry out spoken commands from different seats and answer follow-up requests. Testers check completed actions, including window adjustments and navigation commands. | Everyday | Mar 2025 | Not in cards | |
| RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. | Everyday | Mar 2025 | | Fading |
| Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. | Research frontier | Feb 2025 | 66.7% | System cards only |
| EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. | Everyday | Feb 2025 | 50.5% | Not in cards |
| EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. | Everyday | Feb 2025 | 67.7% | Not in cards |
| EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. | Everyday | Feb 2025 | 58.7% | Not in cards |
| EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. | Everyday | Feb 2025 | 28.5% | Not in cards |
| EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. | Everyday | Feb 2025 | | Not in cards |
| Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. | Expert | Feb 2025 | 31.3 | System cards only |
| Vending-Bench Agents run a simulated vending machine business over long horizons, ordering stock, setting prices, and paying daily fees. Scoring is final net worth in dollars. | Everyday | Feb 2025 | $4,694 | Dropped |
| Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. | Everyday | Jan 2025 | Not in cards | |
| LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. | University | Dec 2024 | Not in cards | |
| Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. | Expert | Dec 2024 | System cards only | |
| VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. | Everyday | Dec 2024 | | Not in cards |
| BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. | Everyday | Nov 2024 | 68.3% | Not in cards |
| PARTNR Agents follow natural language instructions in human-robot collaboration scenes. Tasks require planning and coordination among multiple embodied agents. | University | Nov 2024 | Not in cards | |
| AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. | Expert | Oct 2024 | Not in cards | |
| Embodied Agent Interface Models interpret goals, decompose them into subgoals, sequence actions, and predict state transitions in BEHAVIOR and VirtualHome. The evaluator checks task completion and distinguishes hallucinations, invalid actions, and planning errors. | Everyday | Oct 2024 | 70.3 | Not in cards |
| GLEE Agents bargain, negotiate, or persuade another player through conversation while pursuing the payoff defined by the game. | University | Oct 2024 | 2,595 | Not in cards |
| ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. | Everyday | Oct 2024 | Not in cards | |
| SPA-Bench Agents use Android apps such as Airbnb, Spotify, and WeChat. The grader matches screenshots against task requirements and uses a vision model to resolve uncertain cases. | Everyday | Oct 2024 | 1.35% | Not in cards |
| WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. | Oct 2024 | Not in cards | ||
| DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. | University | Sep 2024 | Not in cards | |
| RoboTwin 1.0 Dual-arm policies train on generated digital-twin demos and are tested on simulated single-arm and dual-arm tasks, with matching real-world data. Success rate. | Everyday | Sep 2024 | Not in cards | |
| VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. | Everyday | Aug 2024 | 36.2% | Not in cards |
| CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. | Expert | Jul 2024 | Not in cards | |
| Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. | Everyday | Jun 2024 | Not in cards | |
| RoboCasa Simulated mobile manipulators and humanoids do 100 kitchen tasks, 25 atomic and 75 composite, across 120 scenes. Scored by success rate. | Everyday | Jun 2024 | | Fading |
| AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. | Expert | May 2024 | 37.8% | Not in cards |
| SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. | Everyday | May 2024 | 80% | In use |
| OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. | Everyday | Apr 2024 | | Dropped |
| BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. | Everyday | Mar 2024 | Not in cards | |
| HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. | Everyday | Mar 2024 | Not in cards | |
| FinBen A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading. | University | Feb 2024 | 0.25 | Not in cards |
| GTBench Models play strategic games against other models or scripted opponents. The benchmark measures their advantage in game rewards. | Everyday | Feb 2024 | 0.13 | Not in cards |
| AgentBoard LLM agents act in 9 partially observable, multi-turn environments that cover embodied tasks, games, web browsing and tool use. Besides final success, AgentBoard reports a progress rate, the share of annotated subgoals the agent reaches. An open toolkit breaks results down by skill. | Everyday | Jan 2024 | 48.9% | Not in cards |
| Economics Arena for Large Language Models Models choose numbers in beauty contests and place sealed bids in auctions. Researchers compare payoffs and measure how strategies change when players see previous rounds. | University | Jan 2024 | 0.95 | Not in cards |
| FinMem Trading Agents choose whether to buy, sell or hold a stock each day. The benchmark measures cumulative return against historical prices. | University | Nov 2023 | 1.63% | Not in cards |
| Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. | University | Nov 2023 | 4.67% | Not in cards |
| SmartPlay Language-model agents play a collection of games designed to test planning, reasoning, memory and interaction abilities. | Expert | Oct 2023 | 0.80 | Not in cards |
| LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. | Everyday | Jun 2023 | | Fading |
| Language-Table A simulated robot moves colored blocks in response to natural language instructions. The benchmark measures success and success weighted by path length across five task families and 696 task variations. | Everyday | Oct 2022 | Not in cards | |
| VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. | Everyday | Oct 2022 | Not in cards | |
| ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. | School | Mar 2022 | 0.15 | Not in cards |
| CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. | Everyday | Dec 2021 | Not in cards | |
| SatNet Agents assign ground antennas and contact times to spacecraft requests across five weeks of NASA Deep Space Network operations. They balance unmet demand across missions. | University | Nov 2021 | Not in cards | |
| TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. | Everyday | Oct 2021 | Not in cards | |
| ALFWorld Agents complete household goals such as putting a washed apple in the fridge in a text world aligned with the ALFRED embodied simulator. Success is whether the goal state is reached. | Everyday | Oct 2020 | 24.6% | Not in cards |
| Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. | Everyday | Oct 2020 | Not in cards | |
| PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. | School | May 2020 | Not in cards | |
| ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. | Everyday | Dec 2019 | Not in cards | |
| Meta-World Robot policies do simulated tabletop manipulation tasks, scored by success rate. Splits test multi-task learning, such as MT10 and MT50, and meta-learning, such as ML45. | Everyday | Oct 2019 | Not in cards | |
| RLBench Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates. | Everyday | Sep 2019 | Not in cards | |
| BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. | Everyday | Oct 2018 | 37.5% | Not in cards |
| Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. | Expert | | System cards only | |
| AnalogCoder-Pro Agents write analog circuit programs and use simulated waveforms to diagnose failures. The authors report the fraction of successful designs and the number of tasks solved. | Expert | 88.4% | Not in cards | |
| BAZAAR Agents buy and sell goods in a simulated competitive market. | University | 7 | Not in cards | |
| Closed-Loop Evaluation of LLM Agents for Embedded Software Development Agents write firmware from engineering briefs and test it on simulated equipment. Hidden checks score device behavior across five control tasks. | University | 100% | Not in cards | |
| CompanyBench Agents complete financial analyst workflows in a simulated bank. Checks compare outputs and database changes with expected results. | Expert | 44.4% | Not in cards | |
| Crest Arcade Micro Mouse Models steer a simulated robot through six daily mazes. The score compares each finish time with the median human time and gives unfinished mazes zero. | Everyday | | Not in cards | |
| Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. | Expert | 0.30% | System cards only | |
| EmbodiedCLUE-VLA (tabletop tasks) Robot policies carry out tasks on a tabletop. | Everyday | | Not in cards | |
| FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. | Everyday | | Not in cards | |
| ICLayout-Bench Agents turn transistor netlists into chip layouts. Design tools check the layout rules and connectivity, then measure electrical performance and area. | Expert | 80.3 | Not in cards | |
| Implicit Intelligence Agents respond to requests whose unstated requirements affect safety, privacy, or accessibility. Labelbox reports the share of scenarios each model passes. | Everyday | 53.2% | Not in cards | |
| Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. | Everyday | | System cards only | |
| Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. | Expert | | System cards only | |
| Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. | Everyday | Not in cards | ||
| Speko LLM Benchmark for Voice Agents v3 Models use simulated support tools to look up orders and issue refunds. The task score gives partial credit when a model stops early without causing harm. | Everyday | | Not in cards | |
| SuperCLUE-XClaw Claw assistant products complete user tasks in Chinese. | Everyday | Not in cards | ||
| UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. | System cards only | |||
| UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. | Expert | 15.6 | System cards only |