Benchmark Almanac

Robotics

Models control a real robot or a simulated one. We count 394 benchmarks here, 9 of them arenas. A benchmark belongs here when environment includes a physical robot or device. Or environment includes a simulator, and output includes actions.

How fast it grows

174 were released since July, more than the 86 in the first half of 2026.

  1. Earlier 17
  2. 2024 H1 11
  3. 2024 H2 15
  4. 2025 H1 32
  5. 2025 H2 41
  6. 2026 H1 86
  7. 2026 H2 174

How much is still open

No model has passed 85% on 166 of the 192 benchmarks here that score in percent, 86% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. Expert Nov 2026 59.5 Not in cards
AbMaze Agents make choices in a simulated antibody optimization experiment under a fixed budget. Recorded assay results determine normalized improvement on 200 test episodes. Expert Nov 2026 Not in cards
ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. Everyday Nov 2026 80.4% Not in cards
ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. Everyday Nov 2026 Not in cards
AstroMemBench Spacecraft agents use information from earlier observations to complete simulated missions. The environment scores task completion, safety, and resource use. University Nov 2026 Not in cards
Crazyrouter AI Trading Arena Models buy, sell, or hold three cryptocurrencies every five minutes using simulated accounts. The grader values portfolios at live quotes and deducts trading fees. University Nov 2026 -7.45% Not in cards
CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. Everyday Nov 2026 0.56 Not in cards
DeGennaro Workplace sample Agents act as an employee’s assistant using 80 simulated workplace tools and anonymized company records. Four tasks grade the final state. Expert Nov 2026 Not in cards
DrivingBench v2 Vision-language models drive a real Tesla through two unseen cone courses with steering, accelerator and brake tools. The score is the furthest progress over three attempts in one conversation, averaged over six trials. Everyday Nov 2026 89.5% Not in cards
Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. University Nov 2026 Not in cards
EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. University Nov 2026 Not in cards
ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. Everyday Nov 2026 Not in cards
FrontierBreach Agents perform simulated enterprise tasks while processing untrusted material. The benchmark checks whether each agent resists instructions that conflict with its assigned task. Expert Nov 2026 2.40% Not in cards
FrontierEE Bench Agents run electrical engineering experiments on real lab hardware through networked instruments, camera photos and tickets to a human operator. Deterministic verifiers check 10 tasks against protected bench records. Expert Nov 2026 36.7% Not in cards
GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. Expert Nov 2026 16.4% Not in cards
GeoGuesser Bench Agents explore offline Street View panoramas to report coordinates, name places, and find objects. Coordinate guesses earn distance points. Other answers require an exact match. Everyday Nov 2026 22.8 Not in cards
GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. Everyday Nov 2026 20.7% Not in cards
GraspBench Robots propose grasps and try to lift objects in MuJoCo. Each attempt must pass motion planning and execution on the chosen robot. Everyday Nov 2026 Not in cards
HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. Everyday Nov 2026 18.8% Not in cards
HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. Everyday Nov 2026 61.9% Not in cards
HW-Bench Agents complete six hardware engineering tasks for a 2.45 GHz antenna using native design and simulation tools. Code checks and calibrated model judges grade outputs against engineer-written criteria. The headline pass rate measures runs that satisfy every critical criterion, averaged across tasks. Expert Nov 2026 67% Not in cards
Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. Expert Nov 2026 83.9% Not in cards
Malicious agentic influence campaigns, 34-point rubric (Anthropic) Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale. Expert Nov 2026 System cards only
ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. Everyday Nov 2026 Not in cards
MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. Everyday Nov 2026 50.4% Not in cards
Parley Labs negotiation tasks Agents negotiate contracts and plan around market changes with scripted counterparties. An offline verifier grades outcomes. Ten public development tasks accompany private scored sets. Expert Nov 2026 Not in cards
Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. Expert Nov 2026 61.1% Not in cards
Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. Expert Nov 2026 0.60 Not in cards
RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. Everyday Nov 2026 51.4% Not in cards
RoboCasa-Mem Robots complete nine household tasks that require memory. Later steps depend on objects the robot saw or actions it took earlier. Everyday Nov 2026 16% Not in cards
RoboColosseum Physical robots execute fresh instructions in paired blind trials. Human evaluators choose the better run or a tie, and comparisons determine ratings for each embodiment. Everyday Nov 2026 Not in cards
RoboDx Agents repair faults in robot navigation software. Hidden driving scenarios test the repair. A separate diagnosis track asks models to identify faults without seeing the code. Expert Nov 2026 100% Not in cards
RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. University Nov 2026 80.6% Not in cards
Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. Everyday Nov 2026 42.9% Not in cards
Sequitur Research Lab Automation Agents run simulated wet-lab automation protocols. Scored out of 100 on how well each experimental phase is carried out, averaged over six runs. Research frontier Nov 2026 74.5 System cards only
ShadowBench Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match. Everyday Nov 2026 48.8 Not in cards
SMB Bench An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule. University Nov 2026 $7,749 Not in cards
UAV Robust Control Hub Agents construct policies for a fictional civilian drone simulator. The benchmark evaluates policies on hidden scenarios and reports separate results under changed conditions. University Nov 2026 Not in cards
VABAR Agents program robots to sort tools, mesh gears, and handle cables in simulation. The harness checks success as object shapes and positions vary. University Nov 2026 Not in cards
WeChat Pay Physical Client Evaluation Agents operate the WeChat Pay client on physical phones. The team scores execution paths, use of supplied knowledge, acceptance checks, and account creation accuracy. University Nov 2026 Not in cards
AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. Everyday Oct 2026 Not in cards
AdaExploreBench Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice. Everyday Oct 2026 70.3% Not in cards
AML Bench (Veris AI) Agents investigate suspicious activity at a simulated bank. A model judge checks whether the recorded service requests and final case disposition satisfy every requirement. Expert Oct 2026 70.8% Not in cards
Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. Expert Oct 2026 Not in cards
Argo-Bench Agents query a simulated food delivery company's warehouse and act on their findings. The grader measures the business consequences of their decisions across 210 tasks. Expert Oct 2026 70.2 Not in cards
AutoVend Agents buy stock and schedule deliveries for 20 simulated vending machines over 180 days. They share one warehouse and pay for each site visit. University Oct 2026 309.4 Not in cards
BARC-Bench Agents handle routine workplace requests and use records to infer privacy and safety constraints. A judge checks whether each attempt completes safely or defers. University Oct 2026 72.7% Not in cards
BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. Everyday Oct 2026 0.56 Not in cards
Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. Expert Oct 2026 37.8% In use
CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. Expert Oct 2026 Not in cards
Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. Expert Oct 2026 $9,514 Not in cards
ClothDojo Robot policies use both hands to flatten and fold garments. The simulator checks success on familiar garments and garments withheld from training. Everyday Oct 2026 44.7% Not in cards
CRM-Agent-Bench Agents handle customer and business requests in a simulated enterprise. Executable checks verify tool arguments, action order, and which records changed across 555 tasks. University Oct 2026 83.6% Not in cards
CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. Expert Oct 2026 87.7% Not in cards
DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. Everyday Oct 2026 40% Not in cards
DexArena Robots use both hands to complete household and industrial tasks. The simulator checks each stage and the final outcome across ten multistage tasks. Everyday Oct 2026 52.1% Not in cards
Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. Everyday Oct 2026 Not in cards
Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. Expert Oct 2026 59.8 Not in cards
Embodied Agent Arena Vision-language agents solve geometry, spatial reasoning, affordance, planning, and manipulation tasks. The 1,000-case evaluation combines 32 established sources with GeoProbe. Everyday Oct 2026 Not in cards
EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. Everyday Oct 2026 52% Not in cards
EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. University Oct 2026 41.4% Not in cards
FlashDexRetarget 50-motion benchmark Retarget 50 human hand-object motions (HOT3D, TACO, OakInk2) to XHand and Sharpa hands in Isaac Sim. Scored by object pose-error success thresholds. Everyday Oct 2026 Not in cards
Hedge Gym Models adjust an options hedge each day over a 21-day price path. The evaluator measures replication error and transaction costs on synthetic or historical markets. Expert Oct 2026 Not in cards
HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. Everyday Oct 2026 49.4% Not in cards
Hy-Nomas-Bench Interactive video world models follow text and camera controls across multiple turns and retain spatial memory. Expert Oct 2026 Not in cards
InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. Everyday Oct 2026 41.7% Not in cards
LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. Everyday Oct 2026 63.3% Not in cards
Learn2Play Bench Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure. Everyday Oct 2026 81.8 Not in cards
LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. Expert Oct 2026 66.5% Not in cards
MAMUBench Agents represent different users who share a limited resource. The benchmark compares their combined outcome with one agent coordinating for everyone. Everyday Oct 2026 Not in cards
Navital Gym Agents complete tasks in six simulated Indian service settings. This port covers UPI Desk, which grades 50 payment and dispute tasks using written rules. Everyday Oct 2026 100% Not in cards
OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. Everyday Oct 2026 Not in cards
PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. Research frontier Oct 2026 14% Not in cards
Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. Everyday Oct 2026 0% Not in cards
PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. Everyday Oct 2026 68.1% Not in cards
Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. Expert Oct 2026 32.7% Not in cards
Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. Everyday Oct 2026 Not in cards
RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. Everyday Oct 2026 19.1% Not in cards
RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. Expert Oct 2026 0.06 Not in cards
RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. Everyday Oct 2026 17% Not in cards
RWE-bench Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement. Expert Oct 2026 62% Not in cards
Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. Expert Oct 2026 $2,949 Not in cards
SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. Expert Oct 2026 100% Not in cards
SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. Expert Oct 2026 Not in cards
SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. Expert Oct 2026 25.1% Not in cards
SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. Expert Oct 2026 46% Not in cards
SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. Expert Oct 2026 27.8% Not in cards
SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. Expert Oct 2026 30.5% Not in cards
Solana Machine Trading Agents write trading bots for hidden market scenarios replayed on offline Solana snapshots. The headline score compares total profit and loss against a reference bot. Expert Oct 2026 6.11 Not in cards
StoreBench Agents manage a simulated apparel store in Medusa. They order inventory, change prices, ship purchases, and process refunds as demand changes. University Oct 2026 0.83 Not in cards
TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. Everyday Oct 2026 Not in cards
TavernBench Agents build a Hearthstone Battlegrounds player through research, simulator updates, and live games. Expert Oct 2026 95.6 Not in cards
The Impossible Refund Assistants pursue a late train refund in a simulated email environment. Separate judges rate deception and other behaviors from the messages. Everyday Oct 2026 Not in cards
Unsatz Agents design experiments and propose equations for an unknown simulated system. Their final program must predict held-out interventions and conditions beyond the original experiments. Expert Oct 2026 60% Not in cards
Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. Expert Sep 2026 0.17% System cards only
ArtifactArena Models design robot bodies, controllers and strategy for a simulated sumo-style arena. Artifacts fight head-to-head and models get Elo ratings. Expert Sep 2026 1,539 Not in cards
CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. Expert Sep 2026 8.60% Not in cards
Cloud Expansion CE80-10 Agents complete a cloud network security exercise. The verifier checks the resulting cloud configuration and observed network behavior. Expert Sep 2026 Not in cards
Cloud Expansion RX00 Agents complete a simulated cloud reconnaissance assessment. The verifier compares the submitted result with the recorded state of the exercise. Expert Sep 2026 Not in cards
CorpBench Work Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly. Everyday Sep 2026 92.7 Not in cards
DolphinBench Agents use long-term memory to complete 600 tool-using tasks for three simulated knowledge workers with roughly 500,000 user-message tokens each. Expert Sep 2026 70.7% Not in cards
DrivingBench Vision-language models drive a real Toyota Corolla through a cone course with up to three attempts in one conversation. Scored on the furthest progress along the course. Everyday Sep 2026 100% Not in cards
DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. Expert Sep 2026 40.0 Not in cards
EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. Research frontier Sep 2026 29.8 Not in cards
ExplorationBench Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help. Research frontier Sep 2026 89% Not in cards
GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. Everyday Sep 2026 80.2 Not in cards
GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Scored by a distance-based GeoGuessr-style reward from 0 to 1. Everyday Sep 2026 0.64 Not in cards
GlyphBench Agents choose actions from text-rendered game boards. The standard suite contains 303 tasks across six game collections and scores agents by cumulative return. University Sep 2026 0.42 Not in cards
HarvestBench Agents drive simulated tractors and decide how much fuel to spend avoiding animals. The benchmark measures how often they hit animals as avoidance costs change. Everyday Sep 2026 0.40% Not in cards
Huayu Awards AI Cockpit Evaluation 2026 In-car assistants interpret requests and operate vehicle features. Tests include planning three-stop trips and adjusting several cabin controls from one request. Everyday Sep 2026 Not in cards
ironbench Agents write firmware for tasks such as serial communication and controller sequencing. The grader checks device behavior and counts tasks solved within the allowed attempts. University Sep 2026 69.2% Not in cards
KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. Expert Sep 2026 29.4% Not in cards
Long-Horizon Medical Agent Benchmark, Veterinary Clinic Ops v2 Agents operate a simulated veterinary clinic through its database API. Each of the 98 tasks checks whether the agent leaves the clinic records in the required state. Expert Sep 2026 100% Not in cards
MANDOS Session 2 Models make process safety decisions across four private industrial scenarios. A model judge assesses each run. The leaderboard reports the share of passing runs. Expert Sep 2026 96.9% Not in cards
McStasBench Agents design neutron scattering instruments. A simulator checks whether their designs reproduce published measurements. The authors score two additional instruments separately. Expert Sep 2026 43.8% Not in cards
metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. Everyday Sep 2026 Not in cards
Nego Eval Agents choose negotiation partners and remember prior encounters across games. The evaluator compares match scores with reference policies on the same seeds. Everyday Sep 2026 Not in cards
OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. Everyday Sep 2026 43.9% Not in cards
OpenRSI Index Isaac Lab PegInsert Reward Search Agents design robot training rewards under a fixed budget. The score is held-out peg-insertion success as a fraction. Research frontier Sep 2026 0.60 Not in cards
Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. Expert Sep 2026 System cards only
PACT (robot arm orchestration) Agents write and revise programs to control a simulated robot arm. The simulator checks whether the arm completes the requested manipulation. Expert Sep 2026 Not in cards
Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. Everyday Sep 2026 36.3% Not in cards
PhoneShell iOS Agent Benchmark Agents operate apps on a physical iPhone to set timers, find books, and complete other everyday tasks. Automated checks read the resulting device state. Everyday Sep 2026 96.6% Not in cards
Prosus Vending Bench v2 Agents run six simulated vending machines across three locations for 30 days. They order stock, set prices, and collect cash. The score measures final bank balance in euros. University Sep 2026 4497.2 Not in cards
Qwen-Audio-Agent Smart Cockpit Benchmark Agents operate simulated vehicle, navigation, music, and weather services. The short suite checks whole cases. The long suite checks tool use on each turn. Everyday Sep 2026 98.8% Not in cards
ReactHuman Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate. Everyday Sep 2026 63.7% Not in cards
REVERSAL-BENCH Reset-free reinforcement learning agents learn manipulation policies across environments with controlled reversibility and recoverability checks. Expert Sep 2026 Not in cards
Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. Everyday Sep 2026 52.5% Not in cards
RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. Everyday Sep 2026 65.7% Not in cards
Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. Everyday Sep 2026 20% System cards only
SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. Expert Sep 2026 45.3% Not in cards
SemiKongBench-100 Agents resolve 100 synthetic semiconductor operating exceptions. The evaluator grades each workflow with deterministic checks. Expert Sep 2026 Not in cards
StationeryBench Robots use both arms to handle common desk supplies. A human grades progress from zero to 100 on five tasks. Everyday Sep 2026 46.0 Not in cards
UK AISI Out of Scope Supply Chain Attack Agents in simulated hard cyber challenges with incidental internet access may attack out-of-scope open source projects. Scored as rate of samples with such attacks. Expert Sep 2026 12% System cards only
VAmoS Pro Bench Voice agents handle several utility billing requests in one call. A model checks the spoken amounts and recorded actions against the requirements for each request. University Sep 2026 40.1% Not in cards
WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. Everyday Sep 2026 42.3% Not in cards
ASI-Bench AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge. Research frontier Aug 2026 55.8 Not in cards
Blobfish Litigation Bench Agents complete 100 tasks in a simulated litigation practice. The grader compares the changes agents make to case records against required outcomes. Expert Aug 2026 Not in cards
CounselBench-100 Agents investigate legal matters and decide which actions the evidence supports. The verifier checks decisions, changes to case records, and advice across 100 synthetic tasks. Expert Aug 2026 Not in cards
DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. Everyday Aug 2026 71.4% Not in cards
E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. University Aug 2026 14.3 Not in cards
EduClaw-Bench Agents tutor a simulated learner over 30 days. The benchmark tracks learning gains, responsiveness, helpfulness, and curriculum design across 55 scenarios. University Aug 2026 Not in cards
EmbodiedCLUE-VLA RoboTwin Evaluation Robot policies perform ten two-handed manipulation tasks under randomized RoboTwin 2.0 Hard conditions. The score averages task success rates over twenty episodes per task. Everyday Aug 2026 Not in cards
FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. Expert Aug 2026 20.6% Not in cards
HEPToolBench 1.2 Models generate inputs for particle physics simulations and inspect the resulting files. The main suite has 28 tasks and a separate extension adds three debugging tasks. Expert Aug 2026 71.4% Not in cards
HM3D-MFMON Agents navigate across floors and find a sequence of objects. Everyday Aug 2026 Not in cards
HWE-bench CAE Agents run engineering simulations, extract numerical results, and report them. A verifier repeats the extraction from the solver output and checks engineering tolerances. Expert Aug 2026 0.90% Not in cards
ITSMBench Agents operate an IT service desk across 53 conversations with simulated colleagues. The verifier grades the resulting database state, including permissions and required actions. University Aug 2026 0.70 Not in cards
LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. Everyday Aug 2026 66.9% Not in cards
MIVAS Bench Voice agents handle calls for a dermatology practice, a law firm, and a retailer. Each call must pass checks on tools, specialist handoffs, and database changes. University Aug 2026 56.5% Not in cards
MobilePA-Bench Agents complete 1,705 mobile planning tasks using tools, persistent memory, reusable skills, and specialist agents. Task-specific checks evaluate tool calls, state changes, and interaction behavior. Additional gates check required memory access and skill loading. Everyday Aug 2026 76.9% Not in cards
S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. Expert Aug 2026 Not in cards
Agentic Trader Agents search for market information and place simulated foreign exchange orders using live prices. Each agent manages its own positions, cash, and margin. University Jul 2026 79.4% Not in cards
ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. Everyday Jul 2026 47.6% Not in cards
BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. Everyday Jul 2026 Not in cards
Bot Analysis Arena Models write trading rules for simulated US stock accounts. Some agents revise their rules after trades. Software tests each revision against the current rules before adoption. University Jul 2026 Not in cards
DexVerse Robot policies perform tabletop manipulation tasks in Isaac Lab, including tool use, bimanual coordination, and sequences with several goals. Task-specific success conditions check completion under different visual settings and robot embodiments. Everyday Jul 2026 34% Not in cards
Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. Expert Jul 2026 97.3% Not in cards
EvoPolicyGym Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves. Expert Jul 2026 0.89 Not in cards
Feishu Office v1 Agents handle 140 Chinese workplace requests in a simulated Feishu workspace. The grader checks the resulting calendars, documents, tables, and approvals. University Jul 2026 Not in cards
Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. Everyday Jul 2026 Not in cards
Game Arena Bargaining Two language models negotiate how to split a pool of items, each with private item values. Each side tries to reach a good deal without giving away its values. Everyday Jul 2026 317 Not in cards
GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. Everyday Jul 2026 Not in cards
Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. Everyday Jul 2026 Not in cards
HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. Everyday Jul 2026 11.1% Not in cards
InfraBench Agents deploy, profile, debug, and optimize model workloads on AMD MI35X GPUs. Task-specific verifiers grade 20 engineering assignments. Expert Jul 2026 Not in cards
MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. Expert Jul 2026 59.5 Not in cards
OptimBench Dispatch Agents route vehicles under capacity and time constraints, then revise their routes after a disruption. University Jul 2026 0.92 Not in cards
RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. Everyday Jul 2026 27.6% In use
SpecDecode RL Agents choose how many tokens to draft during speculative decoding. An offline grader measures speedup over the target model while requiring identical output. Expert Jul 2026 Not in cards
Speko Speech-to-Speech Benchmark (20-turn concierge call) Voice agents try to book a flight, handle a failed booking, and remember a changed date. Eight automated checks score their actions and replies. Everyday Jul 2026 1.00 Not in cards
Task AW IT support Agents complete five support workflows in simulated IT systems. Task checks cover onboarding, offboarding, and cleanup of accounts and assets. University Jul 2026 Not in cards
UAV Operator Models manage simulated drone missions as weather and flight restrictions change. The simulator scores their decisions from logged aircraft and mission state. Expert Jul 2026 Not in cards
VAmoS Bench Voice agents answer simulated bank customers and use tools to manage credit cards. A model checks each call against its transcript and recorded actions. University Jul 2026 62.3% Not in cards
Act2Answer Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control. Everyday Jun 2026 68% Not in cards
AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. Everyday Jun 2026 67.8% Not in cards
ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. Everyday Jun 2026 48.2% Not in cards
CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. Expert Jun 2026 $22,148,357 Not in cards
CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. University Jun 2026 Not in cards
Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. Expert Jun 2026 0.20 Not in cards
DuetBench Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head. Expert Jun 2026 1,035 Not in cards
DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. Everyday Jun 2026 10.6% Not in cards
EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. Everyday Jun 2026 Not in cards
FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. Research frontier Jun 2026 38.1% In use
iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. Everyday Jun 2026 51.9% Not in cards
LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. Expert Jun 2026 0.66 Not in cards
MANDOS Run 003 Models make decisions during seven industrial incidents and four fictional scenarios. A model judge checks their decisions, including whether they seek expert advice. Expert Jun 2026 100% Not in cards
MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. Expert Jun 2026 69.2 Not in cards
MetroLLM-Bench Agents operate a transit kiosk using tools for routes, fares, and service disruptions. The first score checks their actions automatically. The composite also includes model judgments. Everyday Jun 2026 92.0 Not in cards
MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. Everyday Jun 2026 Not in cards
OmniGameArena Models play games while response latency advances the game clock. Everyday Jun 2026 0.47 Not in cards
PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. Expert Jun 2026 95.3% Not in cards
RadioBench Models answer radio engineering questions and generate programs for software-defined radios. The authors measure successful execution, hardware deployment, and reception of the intended signal. University Jun 2026 97% Not in cards
Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. Expert Jun 2026 62.1% Not in cards
RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. Everyday Jun 2026 49.9% Not in cards
Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. Everyday Jun 2026 Not in cards
SentinelBench Agents monitor changing web environments and respond when scripted events make progress possible. Scores measure completion, reaction time and resource use. Expert Jun 2026 Not in cards
SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. Everyday Jun 2026 Not in cards
SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. Everyday Jun 2026 14.4% Not in cards
SpatialWorld Digital Agents complete entertainment tasks in digital environments. The score is the published digital task success rate. Everyday Jun 2026 39% Not in cards
SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. Everyday Jun 2026 14.4% Not in cards
STATE-Bench Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. Expert Jun 2026 58.9% Not in cards
UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. Everyday Jun 2026 55.8% Not in cards
AerialSim Agents fly a simulated drone using camera observations or write a program for the flight. The simulator checks the recorded trajectory against the mission requirements. Expert May 2026 73.2% Not in cards
AgentEscapeBench Agents solve procedural escape rooms using unfamiliar tools. The evaluator checks whether they complete dependent steps and escape. Everyday May 2026 76.9% Not in cards
Agentick Agents solve procedurally generated gridworld tasks involving navigation, planning, reasoning, memory, generalization, and coordination. Scores normalize task performance between random and oracle reference policies. Everyday May 2026 0.63 Not in cards
ASPI Agents complete simulated tasks with incomplete instructions. The authors compare adherence to user intent across matched versions of each task. Everyday May 2026 17.8% Not in cards
BoardGameBench (Homer Quan) Models play seven board game variants against a search engine. The scorer rewards quick wins and gives partial credit for longer losses. Everyday May 2026 264.5 Not in cards
CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. Expert May 2026 54.7% Not in cards
ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. Everyday May 2026 70% Not in cards
HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. Everyday May 2026 Not in cards
NARRA-Gym Agents build interactive stories around a user’s experiences and adapt scenes as the conversation unfolds. Judges rate story quality and user experience separately. Everyday May 2026 2 Not in cards
PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. Expert May 2026 50% System cards only
RoboChallenge Table30 v2 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. Everyday May 2026 0.73% Not in cards
RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. Everyday May 2026 21.5% Not in cards
Science Sim Chemistry Models compare simulated compounds and choose whether to advance or reject them. The evaluator scores decisions against measured reaction outcomes and a budget. University May 2026 Not in cards
TERMS-Bench Agents negotiate the price of a single item with simulated counterparts over up to 10 rounds of alternating offers. Counterparts follow six behavior families across three regimes. A program scores each agent from its logged actions on four separate axes: surplus captured, agreement calibration, belief error about the opponent and procedural violations. Extra modes score merchant profit and a bankroll over many sessions. Everyday May 2026 0.69 Not in cards
TVRBench Agents move through simulated indoor scenes to reproduce a target image's viewpoint. Success requires stopping at exactly the target pose on the simulator's discrete grid. The evaluation split has 500 tasks across 80 scenes. Everyday May 2026 12% Not in cards
WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. May 2026 69.5% Not in cards
Agent Bazaar Agents price goods or trade with sellers in simulated markets. The benchmark measures whether their decisions sustain prices, deter fraud, and preserve welfare and profits. University Apr 2026 Not in cards
AutomationBench Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. University Apr 2026 56.3% In use
Clash of AIs Season 1 Models publish trade entries, targets, and stops before prices move. Clash of AIs simulates fills against market prices and ranks accounts by total equity. University Apr 2026 $283,903 Not in cards
HWE-Bench v1.0 (RTL bug repair) Agents fix bugs in six open-source hardware projects. The benchmark reports the share of fixes that pass repository tests in simulation. Expert Apr 2026 79.9% Not in cards
KellyBench Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth. Expert Apr 2026 -7.90% Not in cards
RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. Everyday Apr 2026 46.2% In use
Spatial-Gym Models find paths that satisfy grid puzzle rules. The evaluation uses 500 SPaRC puzzles in one-shot and stepwise settings, including a setting that allows backtracking. School Apr 2026 Not in cards
UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. Research frontier Apr 2026 80% System cards only
VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. Everyday Apr 2026 43.5% Not in cards
Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. Expert Apr 2026 System cards only
ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. Everyday Mar 2026 100% In use
AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. Expert Mar 2026 76.5% Not in cards
BarterBench Agents trade scarce goods to meet inventory targets. The hub port assigns one model-controlled trader to a marketplace of scripted opponents. University Mar 2026 Not in cards
CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. University Mar 2026 32.3% Not in cards
GUI-CEval Models answer questions about Chinese app screens, locate controls, and choose actions. Agents also complete tasks on physical devices, where the grader checks interface elements. Everyday Mar 2026 61.4% Not in cards
LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. Everyday Mar 2026 71.4% Not in cards
ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. Everyday Mar 2026 Not in cards
PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. Everyday Mar 2026 75.1% Not in cards
RetailBench Agents operate a simulated supermarket for up to 180 days. They choose suppliers, order stock, and set prices as goods age and customers respond. University Mar 2026 $24,351 Not in cards
RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. Everyday Mar 2026 10.4% Not in cards
RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. Everyday Mar 2026 23.9% Not in cards
RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. Everyday Mar 2026 0% Not in cards
AI GameStore Vision-language models play short episodes of 100 games adapted from Apple App Store and Steam top charts. Each game score is divided by the average human score, and models are ranked by the geometric mean. Everyday Feb 2026 12.4% Not in cards
Blicket Test Causal Reasoning Models experiment with a simulated detector to infer which objects activate it. The hub version rewards correct identification and informative experiments. Everyday Feb 2026 0.56 Not in cards
CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. Everyday Feb 2026 Not in cards
EnterpriseBench: CoreCraft Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy. Expert Feb 2026 77.4% In use
EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. Expert Feb 2026 51% Not in cards
MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. Everyday Feb 2026 37.7% Fading
OdysseyArena Agents explore interactive environments built from four transition primitives and must work out the hidden dynamics over long horizons. OdysseyArena-Lite has 120 standard tasks. Expert Feb 2026 44.2% Not in cards
OdysseyArena AI Trading Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials. Expert Feb 2026 23.3% Not in cards
OdysseyArena Energy Dispatch Agents manage energy dispatch under hidden rules. The score is average task success over four trials. Expert Feb 2026 23.3% Not in cards
OdysseyArena Turn on Lights Agents learn hidden switches and turn on lights. The score is average task success over four trials. Expert Feb 2026 28.3% Not in cards
SalesBench (Insurance Sales) Agents find insurance leads, quote plans, and schedule callbacks in a simulated sales pipeline. The evaluator rewards converted monthly premiums and checks compliance with calling restrictions. University Feb 2026 Not in cards
SimulCost Agents tune physics simulations to meet an accuracy target. The authors measure success and computational cost for an initial guess and for repeated attempts with feedback. Expert Feb 2026 80.8% Not in cards
YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. University Feb 2026 $3,445,495 In use
CAR-bench Agents act as an in-car voice assistant that talks with an LLM-simulated driver and uses 58 connected tools for navigation, productivity, charging and vehicle control under set policies. Besides ordinary tasks, hallucination tasks remove a needed tool or fact, and disambiguation tasks need a clarifying question or a lookup. Scores reward consistent success across repeated runs. Everyday Jan 2026 70% Not in cards
MK-Bench Agents schedule jobs when machines break down or orders arrive or disappear. The grader compares each completion time with the best known schedule. University Jan 2026 6.83% Not in cards
RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. Everyday Jan 2026 76.3% Not in cards
ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. Expert Jan 2026 73.8% Fading
Strategy Bench Models compete in six social deduction and strategy game environments. Everyday Jan 2026 Not in cards
TradeRank Arena Models choose positions in cryptocurrencies and US stocks using shared market data and simulated capital. The leaderboard ranks portfolios by return. University Jan 2026 0% Not in cards
CubeBench Agents solve Rubik's Cube tasks across symbolic, visual and partially observed settings to measure spatial reasoning and long-horizon state tracking. Expert Dec 2025 Not in cards
CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Expert Dec 2025 100% In use
REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. Everyday Dec 2025 53.1% Not in cards
SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. Everyday Dec 2025 57.8% Not in cards
Solubility Optimization (anindya) Models choose chemical transformations to improve predicted solubility. The package includes a benchmark runner that measures improvement on evaluation examples. University Dec 2025 Not in cards
SpatialTree Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level. Everyday Dec 2025 58.1% System cards only
Tau2 Bench Verified Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. University Dec 2025 Not in cards
tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. Everyday Dec 2025 82.7% Fading
Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. Expert Nov 2025 0% System cards only
OASIS (agent safety) Agents handle multi-turn requests with different levels of complexity. Evaluators measure the proportion of safe responses. University Nov 2025 95.2% Not in cards
Real-Time Reasoning Gym Models act in games while token generation consumes part of the game clock. Everyday Nov 2025 0.81 Not in cards
Tau2 Bench Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. University Nov 2025 Not in cards
Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. Everyday Nov 2025 $16,843 In use
Vending-Bench Arena Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance. Expert Nov 2025 $15,962 Not in cards
AgentBench FC Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format. Everyday Oct 2025 58.9% Not in cards
BuildArena Agents design machines in the Besiege simulator from written requirements. The simulator tests whether a vehicle reaches its target, how much weight a bridge supports, and how a lifting machine performs. University Oct 2025 45.3% Not in cards
BuilderBench Agents learn to control a simulated robot that builds target block structures through physical interaction and exploration. Expert Oct 2025 98% Not in cards
Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. Everyday Oct 2025 37% Not in cards
LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. Everyday Oct 2025 69.6% Not in cards
LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. Everyday Oct 2025 53% Not in cards
Magentic Marketplace Agents search for sellers, exchange proposals and make purchases. The simulation measures how much value customers receive after paying for their purchases. Everyday Oct 2025 Not in cards
RoboChallenge Fine-tuned policies run 10 real rollouts on each of 30 tabletop tasks across four robot types. Scored by success rate and progress. Everyday Oct 2025 42.7% Not in cards
StockBench Agents trade stocks over a fixed evaluation window. StockBench ranks models by the mean of final return, maximum drawdown and Sortino ratio. The score is the board's rank, so a lower score is better. Expert Oct 2025 1.00 Not in cards
USIM underwater robot tasks Agents follow instructions to navigate, grasp, transport, and track objects underwater. The simulator measures task completion across 700 trials. Everyday Oct 2025 37.6% Not in cards
World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. Everyday Oct 2025 64.8% Not in cards
Automotive-ENV Agents operate climate controls in an Android Automotive OS simulator. A validator checks the screen and vehicle signals after requests such as turning on the defroster. Everyday Sep 2025 49.3% Not in cards
BEHAVIOR Challenge 2025 50 long-horizon household activities on a Galaxea R1 Pro in OmniGibson, trained from 10,000 demos. Scored by Q-score and success. Everyday Sep 2025 Fading
MAPP (Multi-Agent Path Planning) Models move several agents across a grid without collisions. The environment tracks how many agents reach their goals and how closely their routes follow shortest paths. School Sep 2025 70.8% Not in cards
Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. Everyday Sep 2025 42.1% Fading
OceanGym Agents interpret underwater images and sonar, then navigate simulated search and inspection tasks. The simulator scores how closely they reach target locations. University Sep 2025 14.6 Not in cards
VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. Everyday Sep 2025 72.1% Not in cards
DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. Everyday Aug 2025 39% Not in cards
GEN-Bench Agents select job operations as conditions change in a simulated factory. The benchmark compares schedule completion times with the best known result for each instance. University Aug 2025 6.09% Not in cards
LiveTradeBench Agents allocate simulated portfolios across US securities and Polymarket contracts using streamed prices and news. The researchers track portfolio returns over successive trading days. University Aug 2025 12.1% Not in cards
Asper-Bench Agents turn complex user requests into programs that execute multi-step actions in a simulated digital assistant environment. Automated checks compare the resulting state with the requested goal. Expert Jul 2025 Not in cards
Emergent Price-Fixing by LLM Auction Agents Models trade in simulated auctions and communicate with other participants. A judge examines game logs for evidence of price fixing. Everyday Jul 2025 19% Not in cards
Group Binary Search (Riedl) Agents each choose an integer without seeing the other guesses. They adjust after hearing whether the group sum exceeds or falls below a hidden target. Everyday Jul 2025 75% Not in cards
LLM Economist Models set tax rates or choose working hours in a simulated economy. Researchers compare the social welfare achieved under different tax policies. University Jul 2025 Not in cards
OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. Everyday Jul 2025 53.4% Not in cards
RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. Everyday Jul 2025 44.3% Fading
TextQuests Agents play 25 classic interactive fiction games. The score is harmful actions per 100 steps, where lower is better. The board also reports game progress. Everyday Jul 2025 7.00 System cards only
Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. Expert Jun 2025 0% In use
CVDP Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs. Expert Jun 2025 64.1% Not in cards
MCP-SIM Agents turn physics problems into FEniCS simulations. The tasks include heat flow, elastic deformation, and fracture. The authors count solved tasks. University Jun 2025 Not in cards
RoboArena Double-blind pairwise A/B trials of generalist DROID policies on real Franka arms, free-form tasks by volunteer evaluators. Elo-style rating. Everyday Jun 2025 1,608 Not in cards
RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. Everyday Jun 2025 58.3% Not in cards
tau2-bench Dual-control customer-service simulations (retail, airline, telecom) where agent and simulated user both act. Graded on final state. Everyday Jun 2025 98.9% In use
V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. Everyday Jun 2025 Not in cards
Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. Everyday Jun 2025 84.9% Not in cards
WorldPrediction Agents predict and plan in procedural environments with long horizons. Tasks test whether models maintain a useful world model while acting. University Jun 2025 Not in cards
FoamBench Agents create OpenFOAM cases from written specifications. The grader checks whether the solver finishes and whether the final flow field matches a reference simulation. University May 2025 22.5% Not in cards
gg-bench Models play 126 games invented by a language model and receive game-specific outcomes. Everyday May 2025 36.3% Not in cards
KORGym Models play 51 text and visual games designed to test interactive reasoning. Everyday May 2025 0.82 Not in cards
LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. Everyday May 2025 72.4% Fading
Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. Expert May 2025 0.63% System cards only
VideoGameBench Vision-language models play 10 popular 1990s Game Boy and MS-DOS games in real time from raw frames. Scored by the share of each game completed. Expert May 2025 0.48% Not in cards
AI Engineering Lab simulation self-validation Agents predict how a mechanical system should behave, inspect simulation outputs, and decide whether the model is correct. The authors score these decisions with F1. University Apr 2025 0.72 Not in cards
Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. Expert Apr 2025 93.3% System cards only
m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. Everyday Apr 2025 46.5% Dropped
Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. Expert Apr 2025 System cards only
AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. Everyday Mar 2025 40% Not in cards
Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. Everyday Mar 2025 Not in cards
IVISTA Voice Interaction, revised 2023 protocol In-car assistants carry out spoken commands from different seats and answer follow-up requests. Testers check completed actions, including window adjustments and navigation commands. Everyday Mar 2025 Not in cards
RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. Everyday Mar 2025 50% Fading
Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. Research frontier Feb 2025 66.7% System cards only
EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. Everyday Feb 2025 50.5% Not in cards
EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. Everyday Feb 2025 67.7% Not in cards
EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. Everyday Feb 2025 58.7% Not in cards
EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. Everyday Feb 2025 28.5% Not in cards
EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. Everyday Feb 2025 53.7% Not in cards
Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. Expert Feb 2025 31.3 System cards only
Vending-Bench Agents run a simulated vending machine business over long horizons, ordering stock, setting prices, and paying daily fees. Scoring is final net worth in dollars. Everyday Feb 2025 $4,694 Dropped
Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. Everyday Jan 2025 Not in cards
LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. University Dec 2024 Not in cards
Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. Expert Dec 2024 System cards only
VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. Everyday Dec 2024 9.84% Not in cards
BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. Everyday Nov 2024 68.3% Not in cards
PARTNR Agents follow natural language instructions in human-robot collaboration scenes. Tasks require planning and coordination among multiple embodied agents. University Nov 2024 Not in cards
AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. Expert Oct 2024 Not in cards
Embodied Agent Interface Models interpret goals, decompose them into subgoals, sequence actions, and predict state transitions in BEHAVIOR and VirtualHome. The evaluator checks task completion and distinguishes hallucinations, invalid actions, and planning errors. Everyday Oct 2024 70.3 Not in cards
GLEE Agents bargain, negotiate, or persuade another player through conversation while pursuing the payoff defined by the game. University Oct 2024 2,595 Not in cards
ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. Everyday Oct 2024 Not in cards
SPA-Bench Agents use Android apps such as Airbnb, Spotify, and WeChat. The grader matches screenshots against task requirements and uses a vision model to resolve uncertain cases. Everyday Oct 2024 1.35% Not in cards
WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. Oct 2024 Not in cards
DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. University Sep 2024 Not in cards
RoboTwin 1.0 Dual-arm policies train on generated digital-twin demos and are tested on simulated single-arm and dual-arm tasks, with matching real-world data. Success rate. Everyday Sep 2024 Not in cards
VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. Everyday Aug 2024 36.2% Not in cards
CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. Expert Jul 2024 Not in cards
Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. Everyday Jun 2024 Not in cards
RoboCasa Simulated mobile manipulators and humanoids do 100 kitchen tasks, 25 atomic and 75 composite, across 120 scenes. Scored by success rate. Everyday Jun 2024 70.8% Fading
AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. Expert May 2024 37.8% Not in cards
SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. Everyday May 2024 80% In use
OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. Everyday Apr 2024 57.9 Dropped
BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. Everyday Mar 2024 Not in cards
HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. Everyday Mar 2024 Not in cards
FinBen A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading. University Feb 2024 0.25 Not in cards
GTBench Models play strategic games against other models or scripted opponents. The benchmark measures their advantage in game rewards. Everyday Feb 2024 0.13 Not in cards
AgentBoard LLM agents act in 9 partially observable, multi-turn environments that cover embodied tasks, games, web browsing and tool use. Besides final success, AgentBoard reports a progress rate, the share of annotated subgoals the agent reaches. An open toolkit breaks results down by skill. Everyday Jan 2024 48.9% Not in cards
Economics Arena for Large Language Models Models choose numbers in beauty contests and place sealed bids in auctions. Researchers compare payoffs and measure how strategies change when players see previous rounds. University Jan 2024 0.95 Not in cards
FinMem Trading Agents choose whether to buy, sell or hold a stock each day. The benchmark measures cumulative return against historical prices. University Nov 2023 1.63% Not in cards
Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. University Nov 2023 4.67% Not in cards
SmartPlay Language-model agents play a collection of games designed to test planning, reasoning, memory and interaction abilities. Expert Oct 2023 0.80 Not in cards
LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. Everyday Jun 2023 97.2% Fading
Language-Table A simulated robot moves colored blocks in response to natural language instructions. The benchmark measures success and success weighted by path length across five task families and 696 task variations. Everyday Oct 2022 Not in cards
VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. Everyday Oct 2022 Not in cards
ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. School Mar 2022 0.15 Not in cards
CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. Everyday Dec 2021 Not in cards
SatNet Agents assign ground antennas and contact times to spacecraft requests across five weeks of NASA Deep Space Network operations. They balance unmet demand across missions. University Nov 2021 Not in cards
TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. Everyday Oct 2021 Not in cards
ALFWorld Agents complete household goals such as putting a washed apple in the fridge in a text world aligned with the ALFRED embodied simulator. Success is whether the goal state is reached. Everyday Oct 2020 24.6% Not in cards
Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. Everyday Oct 2020 Not in cards
PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. School May 2020 Not in cards
ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. Everyday Dec 2019 Not in cards
Meta-World Robot policies do simulated tabletop manipulation tasks, scored by success rate. Splits test multi-task learning, such as MT10 and MT50, and meta-learning, such as ML45. Everyday Oct 2019 Not in cards
RLBench Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates. Everyday Sep 2019 Not in cards
BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. Everyday Oct 2018 37.5% Not in cards
Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. Expert 0.49 System cards only
AnalogCoder-Pro Agents write analog circuit programs and use simulated waveforms to diagnose failures. The authors report the fraction of successful designs and the number of tasks solved. Expert 88.4% Not in cards
BAZAAR Agents buy and sell goods in a simulated competitive market. University 7 Not in cards
Closed-Loop Evaluation of LLM Agents for Embedded Software Development Agents write firmware from engineering briefs and test it on simulated equipment. Hidden checks score device behavior across five control tasks. University 100% Not in cards
CompanyBench Agents complete financial analyst workflows in a simulated bank. Checks compare outputs and database changes with expected results. Expert 44.4% Not in cards
Crest Arcade Micro Mouse Models steer a simulated robot through six daily mazes. The score compares each finish time with the median human time and gives unfinished mazes zero. Everyday 89.3 Not in cards
Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. Expert 0.30% System cards only
EmbodiedCLUE-VLA (tabletop tasks) Robot policies carry out tasks on a tabletop. Everyday 49.3 Not in cards
FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. Everyday $215,319 Not in cards
ICLayout-Bench Agents turn transistor netlists into chip layouts. Design tools check the layout rules and connectivity, then measure electrical performance and area. Expert 80.3 Not in cards
Implicit Intelligence Agents respond to requests whose unstated requirements affect safety, privacy, or accessibility. Labelbox reports the share of scenarios each model passes. Everyday 53.2% Not in cards
Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. Everyday 93.8% System cards only
Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. Expert 8.50% System cards only
Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. Everyday Not in cards
Speko LLM Benchmark for Voice Agents v3 Models use simulated support tools to look up orders and issue refunds. The task score gives partial credit when a model stops early without causing harm. Everyday 94.7% Not in cards
SuperCLUE-XClaw Claw assistant products complete user tasks in Chinese. Everyday Not in cards
UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. System cards only
UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. Expert 15.6 System cards only