New benchmarks and models

205 questions on how well an agent understands its company, built from a month of real Slack history: who wrote an anonymized message, whom a teammate tagged, and who sent each reply. Agents search a replayed sandbox workspace using only messages from before the answer.

Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior.

Models get a known-vulnerable snapshot of an open-source repository and the list of files its fix changed, then report security bugs in one attempt. A model judge counts a success when a report has the advisory's root cause. 1,829 advisories from October 2026.

Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions.

Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior.

Monitors replayed agentic conversations for harmful or unauthorized actions. Scored by the rate above a severity threshold.

Artificial Analysis' implementation of HeyGen's Code2Video Bench: models write a HyperFrames HTML composition for each of 114 product and brand video briefs. A model judge compares two rendered videos at a time on five axes, and the comparisons become Elo ratings.

Agentic environments where a task can be finished honestly or by crossing a declared line, such as shipping insecure code, leaking a secret, harming a third party through a shared service or leaving the scope of a penetration test. Reports how often each line is crossed.

Replays previously found cyber jailbreak attacks against fixed ExploitBench tasks with real-time blocking turned off. Scored by mean task success rate under attack.

Reflection AI's in-house safety suite of single-turn, multi-turn, jailbreak and agentic requests. Reports the share of harmful requests refused or redirected and the share of harmless requests answered.

Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDojo. Each episode pairs a user task with an instruction planted in a tool result. Reports attack success rate and utility.

Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDyn. Reports attack success rate and utility.

Listeners rate generated speech on reliability and expressiveness. The benchmark reports an overall quality index.

Listeners rate generated voices for English voice design, multilingual control, accents and voice qualities.

People hold live conversations with two hidden speech-to-speech agents on the same scenario, some with tool calls, then pick the one they preferred. Votes are fit to an Elo rating, and the arena also tracks task success.

Speech transcription systems are scored for semantic accuracy with and without free-form context.

150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness.

Speech transcription systems are scored by word error rate on real-world recordings across multiple languages.

1,114 five-choice questions on 279 complex videos. Each question needs evidence from several separate moments, so no single frame answers it.

Measures speculative decoding across varied prompts and domains. Reports the average acceptance length of draft tokens, so higher means faster generation at the same output.

Human-annotated questions on video-based spatial intelligence, organized in four levels from perception to planning and prediction.

Spatial reasoning questions on outdoor pedestrian-view videos with metric 3D ground truth from synchronized stereo cameras, LiDAR and GPS. 8,736 questions range from relative direction to metric and motion estimates.

Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level.

Video questions about dynamic scenes where both the observer and objects move, testing whether models separate camera motion from object motion.

Spatial perception and reasoning questions built from ScanNet scenes. A single-image set covers distance, size and direction, and a multi-view set adds counting across views. Answers are numbers or multiple choice.

Video questions about how spatial relationships change over time in 3D scenes, over 138,000 pairs across five tasks.

Spatial perception and reasoning questions over single and multiple views of indoor 3D scenes, from depth and distance estimates to object relations and spatial imagination.

A large benchmark for 3D object detection from a single image, built by unifying indoor and outdoor datasets such as SUN RGB-D, ARKitScenes, Hypersim, Objectron, KITTI and nuScenes. Vision-language models answer its 3D grounding questions.

Agents audit and patch vulnerabilities in 250 codebases. The score is pass@1 under deterministic verification.

Agent teams formalize Fermat's Last Theorem in Lean. Scores measure the weighted proportion of 60 accepted intermediate theorems.

Measures reward hacking: how often an agent games its grader instead of doing the task. Lower is better.

Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale.

Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards.

Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit.

Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit.

Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit.

Browser agents face adaptive prompt injections across 1,024 tasks. The score is attack success within 64 attempts.

Image generation systems produce infographics and an automated rater scores factuality.

Resolution rate on private enterprise coding tasks, averaged across five runs per task.

Artificial Analysis's combined score for speech-to-speech models.

Models interpret 161 professional life-science visual artifacts. The score is judged accuracy averaged across three rollouts.

Models make decisions over images and screens. The score is the published equal-weight mean of public and private vision scores.

The HappyWorld arena publishes a separate embodied vote pool and Elo ranking.

The HappyWorld arena publishes a separate spatial reconstruction vote pool and Elo ranking.

Models complete computer-use tasks while avoiding unsafe actions.

Corrected PHYSICS test problems with answer-level accuracy. The complete verified test set.

Speech transcription systems are scored by average word error rate across the Artificial Analysis evaluation set.

Scenario reliability across three repeated calls (Pass³).

Published composite score for production voice agents.

Real caller reliability across three repeated calls (Pass³).

Overall word error rate across the published audio clips.

Round-trip word error rate averaged across the two transcribers, as displayed by Cekura.

Mean end-of-turn score across six line conditions, as published by Cekura.

Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours.

Agents complete 127 tasks through version-pinned MCP servers. Stabilized verifiers score single-run task success.

Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them.

Models speak at the correct moment and give the expected semantic response during simultaneous speech.

Models answer spoken questions when a streamed video first reveals the answer.

Models count repeated actions in streamed video and are scored by distance from the true count.

Agents complete enterprise tasks in travel, customer support, and shopping. The score is overall pass@1 for this track and version.

Models initiate speech at specified times while producing the requested content.

Agents complete 100 workspace tasks. The leaderboard reports the percentage of 1,856 rubric checks passed, with separate runs for each model and harness.

Models review legal documents and receive rubric scores for substantive accuracy and editing quality.

Speech models transcribe call-like audio. The score is mean word error percentage over the published pooled 24-hour window.

Speech models synthesize customer-service prompts. The score is median time to first audible sample in milliseconds over the published 24-hour window.

Agents complete realistic tasks through command-line tools. The benchmark measures task success across search, execution, adaptability, ambiguity, and time constraints.

Multimodal agents retrieve and reason over image information for image mining tasks.

Multimodal agents search and answer questions that require visual and textual evidence.

Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them.

Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them.

The public 30-task Lite leaderboard ranks passed-test percentages and reports fully resolved-task percentages separately.

Google compares edited images side by side and converts human preferences into Elo ratings.

Google compares generated images side by side and converts human preferences into Elo ratings.

The current official results table evaluates the Pro release. Original VBVR results must remain separate.

Models answer expert financial questions grounded in business data. The score is average accuracy on eight offline tasks.

Image models render text in images across regions and languages. CVTG-2K reports normalized edit distance, CLIPScore and word accuracy.

Research agents prepare reports. The score is average report quality on the original evaluation tasks.

Image models render long text in images. The benchmark reports separate English and Chinese scores.

Image models generate images from prompts with English and Chinese evaluation splits.

Image models generate short and long images scored on overall image quality.

Agents solve 179 algorithmic tasks under revised scoring that rewards solutions beyond the reference. The score is the official average task score.

Code agents compete in Gomoku. The public competition board reports Elo Rating for each agent configuration.

Code agents compete in OIBench-5. The public competition board reports Time AUC for each agent configuration.

Code agents compete in Pommerman. The public competition board reports Elo Rating for each agent configuration.

Code agents compete in SWE-Perf-10. The public competition board reports Time Ratio for each agent configuration.

Code agents compete in Tetris. The public competition board reports Score for each agent configuration.

Code agents compete in Texas Holdem. The public competition board reports Elo Rating for each agent configuration.

Agents resolve held-out software issues from the private SWE-Bench Pro split. The leaderboard reports the percentage of resolved tasks.

Models turn image prompts into voxel structures. Human comparisons determine Glicko-2 ratings in a separate image-prompt vote pool.

Models generate interactive visual code artifacts for 1,825 tasks. An MLLM judge scores rendered screenshots and code against a per-task checklist.

Models generate music with vocals. Listeners compare anonymous tracks, and separate vocal preference votes determine the Arena score.

Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score.

Coding agents solve 25 self-contained TypeScript exercises from Exercism. The score is the percentage of exercises that pass their tests.

Agents implement research paper code. The Code-Dev variant evaluates coding separately from full research replication.

Models answer the original 268-question cleaned GSM8K subset. The live matrix reports each model configuration separately.

Models retrieve relevant documents from multimodal document collections.

Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite.

Systems translate text in natural images across multiple language directions.

Models edit images from user instructions. Anonymous human comparisons determine Elo in this independent vote pool.

Models generate images from text prompts. Anonymous human comparisons determine Elo in this independent vote pool.

Models generate videos from text prompts. Anonymous human comparisons determine Elo in this independent vote pool.

Models answer foundational audio understanding questions. The score is the official average accuracy across nineteen tasks.

Models solve financial evaluation tasks. The score is the official 6.0 total score.

Models detect 3D objects in nuScenes sensor data. The official test leaderboard ranks submissions by nuScenes Detection Score.

Models retrieve cooking videos from text queries in the YouCook2 collection.

Systems select the correct story ending on the Winter 2018 test set. CodaLab reports percentage accuracy under submission identities.

Models retrieve videos from text queries in the MSR-VTT collection.

Models create charts. Pairwise votes determine the published Arena rating.

Models create videos through code. Pairwise votes determine the published Arena rating.

Users compare anonymous multimodal model answers and vote for the better response. A Bradley-Terry model estimates ratings from these votes.

Models rewrite text in corporate language. Pairwise votes determine the published Arena rating.

Coding agents build apps, games, slides and spreadsheets from user prompts. Users compare the results, and votes from every agent board form one rating.

Agents build a 3D scene in Blender from a user prompt. Users compare the rendered scenes and vote for the better one.

Agents write code that renders a video from a user prompt. Users compare the two videos and vote for the better one.

Image models make a graphic design piece from a user prompt. Users compare two anonymous designs and vote for the better one.

Image models design a logo from a user prompt. Users compare two anonymous logos and vote for the better one.

AI app builders make a mobile app from the same user prompt. Users compare the apps and vote for the better one.

Agents build a PDF document from a user prompt. Users compare the two documents and vote for the better one.

Models retrieve matching documents and images using the DOCCI corpus.

Models explain concepts in simple language. Pairwise votes determine the published Arena rating.

Models furnish hand-drawn room sketches. People compare the layouts and vote for the better floorplan.

Models prove statements in the FormalMATH Lite subset. The board reports proof success at the stated sampling budget.

Image generation systems are compared by human side-by-side preference on text-to-image prompts.

Image generation systems are evaluated on subject-focused educational tasks with expert ratings.

Models write jokes. Pairwise votes determine the published Arena rating.

Models create landing pages. Pairwise votes determine the published Arena rating.

Embedding models complete the MIEB Lite task set. The official API reports the mean across task types.

Models retrieve latent associations for the ten hardest NoLiMa question-needle pairs. Results are reported separately at four context lengths.

Models write opening lines for stories. Pairwise votes determine the published Arena rating.

Models generate speech. Listeners compare outputs, and pairwise votes determine Elo ratings.

Agents solve unpublished research problems written by professional scientists. Most problems concern theoretical physics, with some partial credit.

Models produce audio responses to user prompts. Anonymous human comparisons determine the published rating in this separate vote pool.

Models produce image responses to user prompts. Anonymous human comparisons determine the published rating in this separate vote pool.

Models produce text responses to user prompts. Anonymous human comparisons determine the published rating in this separate vote pool.

Models retrieve matching text and images using the TextCaps corpus.

Models generate speech in Arabic MSA. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in Bahasa. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in Brazilian Portuguese. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in French. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in German. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in Hindi. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in Indian English. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in Japanese. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in Mexican Spanish. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in Tagalog. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in US English. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Models generate speech in Vietnamese. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings.

Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed.

Artificial Analysis runs agents on Harvey's Legal Agent Benchmark tasks in an offline sandbox, where they write legal work products such as memos and redlines from case documents. Judges grade each deliverable against a task rubric, and the headline is the share of rubric criteria passed.

Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit.

Models predict which genes change after drugs, gene knockdowns, and immune signals across four prediction tracks containing 60, 60, 60, and 27 cases. The overall score averages perturbation-specific F1 across those tracks on a zero-to-one scale.

Models select decisions from short voice-call clips and conversation history across 4,867 questions, 1,081 clips, and 32 tasks. Human reference labels determine accuracy. The default board ranks by DECIDE Balance Score, which combines accuracy, cost, and latency on a zero-to-100 scale.

Agents create editable Blender objects and scenes or executable Unreal Engine worlds across 145 tasks and nine task families. Model judgments and native project checks assess appearance, geometry, requirements, and interactions. The overall score weights object, scene, and world scores by 30%, 30%, and 40%.

Agents complete 83 professional tasks across 36 occupations, using source files to produce work such as documents, spreadsheets, and code. The Joules-to-GDP framework compares outputs with expert deliverables through a blinded model judge. Task success measures the share of wins and ties against the expert.

Agents complete six hardware engineering tasks for a 2.45 GHz antenna using native design and simulation tools. Code checks and calibrated model judges grade outputs against engineer-written criteria. The headline pass rate measures runs that satisfy every critical criterion, averaged across tasks.

Models answer personal finance requests and paired education questions. Moneda’s default board scores paired pass rate on 50 EU personal requests with education twins.

Models answer finance questions across seven separate benchmarks. AdviceBench, CiteBench, FilingsBench, GroundBench and PortfolioBench have their own records. KnowledgeBench and MarketBench have no published results yet. Moneda publishes no combined score.

Models write company research notes with linked sources. Moneda scores the weighted share of linked claims that the cited pages support across 100 notes.

Models answer questions about SEC filings without receiving documents. Moneda scores accuracy on the private questions using filing data and judges for segment questions.

Models answer live questions about market figures. Moneda’s default board reports the share of correct answers against reference data, with incorrect answers reported separately.

Models solve hypothetical portfolio tasks involving rebalancing, risk, position sizing, and constraints. Moneda’s default board scores accuracy on 150 tasks with fixed answers.

Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits.

Models classify land instruments, extract lease terms, and distinguish petroleum terms in three separate benchmarks. Classify and Extract have their own records. Define publishes three measures without one headline score.

Models identify the legal act performed by each of 66 synthetic land instruments. Collide scores label accuracy under open and closed prompt protocols.

Models extract ten shared lease fields from 628 synthetic instruments. Collide divides correct field decisions by correct, omitted, and wrong decisions, excluding unscored fields.

Agents place supplied CAD parts into 16 assemblies. The score is the published mean assembly geometry score.

Agents build 39 CAD parts from engineering drawings. The score is the published mean geometry score.

Agents complete workplace workflows while attackers add misleading records to an emulated company with 11 services. The benchmark pairs six workflow types with harmful injected objectives across 34 tasks. The headline here measures single-query attack success as a percentage, where lower is better.

Models answer explicit requests during audio and video streams. The score is the published average across tasks T1 through T7.

Models provide timely proactive assistance during audio and video streams. The score is the published T8 assistance recall.

Models respond to simulated teens in health advice scenarios. The score is the published conversation failure rate.

Models respond to simulated teens in relationship boundary scenarios. The score is the published conversation failure rate.

Models respond to simulated teens in urgent safety scenarios. The score is the published conversation failure rate.

Models receive 144 benign biology prompts. The score measures the percentage correctly answered.

Models receive 398 dangerous biology prompts. The score measures the percentage correctly refused.

Models value companies using financial evidence and calculations. The score is the published CorporateValuation percentage.

Models write finance research reports. The score is the published DeepResearch percentage.

Models retrieve and verify live financial facts. The score is the published LiveSearch percentage.

Agents repair a reasoning playbook under a fixed budget. The score is held-out Formula exact-answer accuracy as a fraction.

Agents design robot training rewards under a fixed budget. The score is held-out peg-insertion success as a fraction.

Agents improve a decision model under a fixed budget. The score is the published exponential of negative macro log loss.

Agents adapt mathematical reasoning examples under a fixed training budget. The score is accuracy across five math benchmarks.

Agents design one optimizer for six model sizes. The score is the published complete six-rung reward relative to AdamH.

Agents optimize a sparse-attention prefill operator under a fixed budget. The score is speedup against the paired reference run.

Agents tune video inputs and inference for a frozen Molmo2 model. The score is held-out spatiotemporal pointing soft F1.

Agents select interaction context for a frozen browser model. The score is macro-averaged browser-action prediction quality.

Agents adjust retrieval training data schedules. The score measures reasoning retrieval with a penalty for general retrieval regression.

Agents improve speech serving while preserving quality. The score is the official speedup reward for the archived continuation protocol.

Agents choose safe interface actions on the Desktop test set. The published safety score combines task success and attack resistance.

Agents choose safe interface actions on the Mobile test set. The published safety score combines task success and attack resistance.

Models complete separately scored speech recognition, speech generation, voice-agent, real-call, turn-taking and voice-quality benchmarks. Cekura publishes no combined score.

Agents solve questions about changing user preferences. The score is the published accuracy across versioned tasks.

Agents solve software tasks that change across versions. The score is the published accuracy across versioned tasks.

Agents solve terminal tasks that change across versions. The score is the published accuracy across versioned tasks.

Agents complete entertainment tasks in digital environments. The score is the published digital task success rate.

Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories.

Models generate 3D assets through code. Human preference votes determine the published Bradley-Terry Elo rating.

Models prepare training data for the finance domain. The score is the published downstream average after backbone training.

Models prepare training data for the general domain. The score is the published downstream average after backbone training.

Models prepare training data for the law domain. The score is the published downstream average after backbone training.

Models prepare training data for the math domain. The score is the published downstream average after backbone training.

Models prepare training data for the medical domain. The score is the published downstream average after backbone training.

Models prepare training data for the science domain. The score is the published downstream average after backbone training.

Agents rebuild 200 programs from only their compiled binaries and docs. The score is the share of tasks whose rebuilt program passes at least 95% of the hidden behavioral tests.

Agents rebuild 200 programs from only their compiled binaries and docs. The score is the share of hidden behavioral tests the rebuilt program passes, averaged over tasks.

Agents navigate local copies of 153 websites to search, sign in, book services, and change stored state. The suite contains 3,518 tasks, including 306 verified test tasks and 2,288 training tasks. The test score measures the percentage of tasks passed by scripted verifiers.

Models edit text in real images in the full TextEdit test set. The score is the published judged average.

Models edit text in virtual images in the full TextEdit test set. The score is the published judged average.

Models answer the text-only questions of Humanity's Last Exam after expert verification and revision. The Full set keeps the questions experts marked uncertain, and the score is pass@1 accuracy.

Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials.

Agents manage energy dispatch under hidden rules. The score is average task success over four trials.

Agents operate repositories with changing hidden rules. The score is average task success over four trials.

Agents learn hidden switches and turn on lights. The score is average task success over four trials.

Models read values and units from real instrument images. The score is the published overall measurement accuracy.

Models read values and units from generated instrument images. The score is the published overall measurement accuracy.

Models solve spatial location questions in the RefSpatial expansion set. The score is the published overall accuracy.

Models solve spatial placement questions in the RefSpatial expansion set. The score is the published overall accuracy.

Agents search for financial information about global assets across three task types. The score is the published average accuracy for the Global subset.

Agents search for financial information about Greater China assets across three task types. The score is the published average accuracy for the Greater China subset.

Models solve 47 hard final-answer problems that MathArena selected from 2025 competitions, where the best models scored about 50% at selection. Scored by accuracy over repeated attempts.

Agents train models for 30 of MLE-bench's Kaggle-style competitions. The score is the share of runs that earn a bronze medal or better.

Models transcribe speech from Moonshot internal subset 1. The score is the published word error rate.

Models transcribe speech from Moonshot internal subset 2. The score is the published word error rate.

Models follow complex instructions with feedback for up to 20 turns. The score measures fully usable responses as a percentage.

Models predict the next desktop interface action. The score is the published step success rate.

Models locate interface elements from instructions. The score is the published overall grounding accuracy.

Models identify interface layout regions. The score is the published intersection over union percentage.

Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.

Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups.

Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage.

Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage.

Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage.

Models answer driving behavior questions. The score is the published GPT evaluation score for this task family.

Models answer driving perception questions. The score is the published GPT evaluation score for this task family.

Models answer driving planning questions. The score is the published GPT evaluation score for this task family.

Models answer driving prediction questions. The score is the published GPT evaluation score for this task family.

Models respond to spoken AlpacaEval instructions. The score is the published judge quality score.

Models answer spoken Llama Questions. The score is the published accuracy.

Models answer spoken reasoning questions. The score is the published accuracy.

Models answer spoken TriviaQA questions. The score is the published judge quality score.

Models answer spoken Web Questions. The score is the published judge quality score.

Agents complete 417 app tasks that require Amazon or Gmail APIs held out from training. The score is the percentage passing every task test.

Agents complete 168 everyday app tasks from familiar applications by calling APIs through code. The score is the percentage passing every task test.

Agents help simulated airline customers through tools while following airline policies on the original tasks. The score is pass^1 as a percentage.

Agents help simulated retail customers through tools while following store policies on the original tasks. The score is pass^1 as a percentage.

Models complete Chinese financial language tasks. The score is the published average of the application task metrics.

Models answer Chinese financial qualification questions. The score is answer accuracy across the knowledge test.

Models generate acceptance tests for repositories. The score is the published oracle-test percentage.

Models prepare repository environments. The score is the percentage that pass the example usage test.

Models implement repository code. The score is the acceptance-test pass rate, with unit-test pass rate as an auxiliary metric.

Models generate unit tests for repositories. The score is the published oracle-test percentage.

Models use a Python interpreter for general tasks. The score is the percentage of generated programs that execute.

Models use a Python interpreter to solve math problems. The score is execution accuracy on the Math task set.

Models answer multiple-choice questions comparing low-level attributes across image pairs. The score is overall answer accuracy in the A1-Pair table.

Models answer multiple-choice questions about low-level attributes of individual images. The score is overall answer accuracy in the A1-Single table.

Models answer questions and complete other long-context tasks in the original Chinese headline table. The score is its published average on a percentage scale.

Models answer questions and complete other long-context tasks in the original English headline table. The score is its published average on a percentage scale.

Models identify chemical entities in biomedical text. The score is test-set F1.

Models extract chemical and disease interactions from text. The score is test-set F1.

Models extract chemical and protein interactions from text. The score is test-set F1.

Models generate molecules from descriptions. The score is exact-match accuracy on the molecule design test set.

Models predict reaction products from reactants. The score is exact-match accuracy on the forward reaction test set.

Models answer biomedical multiple-choice questions. The score is test-set accuracy.

Models predict reaction reagents. The score is exact-match accuracy on the reagent prediction test set.

Models predict reactants for target products. The score is exact-match accuracy on the retrosynthesis test set.

Models answer biomedical true or false questions. The score is test-set accuracy.

Models write biographies of people, and each output is split into atomic facts. The score is the share of facts that a knowledge source supports, as the paper defines FActScore.

Models judge whether image captions correctly describe spatial relations on the randomly partitioned test set. The score is accuracy over 2,195 test examples.

Models judge spatial captions on a test set whose object concepts do not appear in training. The score is accuracy over 616 test examples.

Models transcribe spontaneous Mandarin speech from real meetings in the Test_Meeting set. The baseline score is mixture error rate, counting Mandarin characters and English words.

Models transcribe Mandarin internet speech from the Test_Net set. The baseline score is mixture error rate, counting Mandarin characters and English words.

Models label premise and hypothesis pairs from the first adversarial test round as entailment, neutral, or contradiction. The score is test accuracy.

Models label premise and hypothesis pairs from the second adversarial test round as entailment, neutral, or contradiction. The score is test accuracy.

Models label premise and hypothesis pairs from the third adversarial test round as entailment, neutral, or contradiction. The score is test accuracy.

Models locate cooking steps and describe each step. The score is CIDEr for the complete dense video description task.

Models play the social deduction game Blood on the Clocktower (the Trouble Brewing script) against each other, with a scripted Storyteller. One model takes every good seat and another every evil seat, so each game tests one model's detection of the Demon against another's deception. A Bradley–Terry fit turns 386 games into Elo ratings.

Agents do private equity and investment banking work on 21 anonymized real deals: valuation, due diligence, underwriting and financial modeling. Each task comes with a data room of about 19 files and would take a professional about 27 hours. Rubrics of about 84 criteria grade the output, the workflow and the judgment behind each assumption.

Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match.

Models write DAX, the formula language of Power BI, for 94 tasks on understanding, writing and reasoning over the Contoso sample model. Each answer runs inside real DAX queries in several report contexts, such as detail rows, subtotals, the grand total and slicers. Harder tasks earn more points, with bonuses for best practices and efficient code.

A model rebuilds an object in 3D from eight photos by placing up to 300 Gaussians itself through tool calls, with no optimizer. Half of the 30 tasks start from an empty scene, and half ask it to reassemble a scrambled fit. SplatScore compares renders from eight angles the model never sees, with an empty scene at 0 and gradient descent at 100.

Ten models play the social deduction game Blood on the Clocktower together, one seat each, over 300 ten-player games of the Trouble Brewing script. Each model plays 210 games on the good team and 90 on the evil team. The score is the share of games its team won.

Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers.

Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly.

Models write regulatory affairs deliverables for drugs, biologics and medical devices, such as change assessment memos, dossier inconsistency logs and agency letters. The 55 tasks span five families: reporting categories, Module 2/3 dossier consistency, missing evidence, submissions and correspondence, and device vigilance. A model judge grades each answer against a task rubric, and a task passes only when every rubric line is met and no penalty fires.

Turn-taking systems listen to two-person conversations and must detect when a speaker's turn ends and when the other speaker interrupts. The test set has 116 English conversations in six interaction styles, each labeled by three annotators. Systems rank by recall, and a system whose false-positive rate exceeds 0.15 ranks below every qualifying system. Latency after each event is also reported.

Multimodal agents build interactive 3D worlds from a user request. They infer intent, plan the scene layout, call 3D tools for asset retrieval, editing and rendering, and revise from rendered feedback. The benchmark has 2,616 3D assets, 323 human-annotated seed worlds and 6,828 queries. Verified queries have ground truth, and the rest are scored with rubrics.

Models get the rules of a strategy game and write a program that plays it. DuelLab checks and builds each program, then runs head-to-head matches between programs from different models. Match outcomes become per-game scores from 0 to 100 and a Bradley-Terry ranking, with no human or model judge.

Coding agents write GPU kernels for real model workloads, such as fused MoE layers, paged attention and FP8 matrix multiplies. The site groups tasks into suites: Mega (one decode task), CUDA (4 tasks) and Hard (6 tasks). A run counts only if it passes an audit of correctness. Scores are speedup over a baseline or percent of the hardware roofline.

Coding agents do 39 realistic React tasks in two tracks. Write React tasks come from merged pull requests and are checked by hidden behavior tests. Fix React tasks ask the agent to remove React performance, accessibility and quality issues. A task passes only if the tests pass and a pinned React Doctor scan finds no new graded issue.

Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control.

Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs.

Memory agents serve several people who share one memory pool in hospitals, offices, schools and homes. Each long episode checks three things: answering legitimate requests after state updates, refusing information a person is not allowed to see, and forgetting records after an explicit deletion request. Hidden checkpoints and leak targets let a structured judge score each response.

Coding agents work through 109 repository tasks rebuilt from real multi-turn sessions between users and coding agents. A simulated user replays the original user's goals and gives feedback when the agent needs it. Scores count tasks solved and the number of corrections the user had to make.

Agents predict the cascade of changes that one action sets off in an enterprise system, where business rules differ between deployments. The environments are synthetic and follow the World of Workflows method. Tests include shifts in the system's rules to check whether a model reads the active configuration or relies on what it learned in training.

Multimodal models answer questions about long PDFs and must cite the page region that supports each answer with a bounding box. The set has 1,897 questions over 711 documents in seven domains and two languages, averaging 40.6 pages per document. Strict Attributed Accuracy counts an answer only when both the answer and the cited region are correct.

Models forecast scientific progress across 4,760 events in many fields, using only knowledge from before a set cutoff date. Tasks ask whether an advance is feasible, how it would work, how to design a solution and when it will happen. The benchmark also compares events before and after each model's training cutoff.

Language models act as simulators of tool environments for agent training. Given a trajectory of agent actions, the model predicts the next environment state and feedback. Labels come from real executors, so the score checks whether simulated states match what the real environment would return.

Agents run inside execution harnesses such as Claude Code, Codex and OpenClaw on 210 tasks across eight domains, from finance to software engineering, with safety rules built in. The audit reads the full trajectory through channels the agent cannot change. It scores whether the agent stays inside its permission and information-flow policies, completes checkpoints, and keeps both under prompt injection, vague goals and tool errors.

Models translate text while following explicit instructions, such as keeping JSON or HTML structure, using a glossary, using provided context and matching a register. It covers seven languages with 4,506 single-constraint and 2,838 multi-constraint items. Deterministic checkers verify some constraints, and a rubric-based model judge scores the rest.

Computer-use agents complete 1,000 desktop tasks across 33 real applications, including browsers, office tools, creative software, development environments and messaging apps. App-specific verifiers inspect the final application state, and the harness gives partial credit from the recorded trajectory.

Memory systems for long-running agents decide what to store from years of interactions across many domains and user personas. Later questions check whether the system kept the context that matters for each user. The benchmark tests whether memory policies can adapt to the person instead of storing everything the same way.

Agents write and run code for 87 quantitative finance tasks in a network-isolated sandbox with real data. Tasks cover derivatives pricing, fixed income, credit, risk, factor research, backtesting and market microstructure. More than 3,300 assertions check numeric outputs against finance rules such as no-arbitrage bounds and put-call parity. Scored by pass@1 over three runs.

Agents with long-term memory must notice when a later observation quietly makes an earlier memory false. The benchmark has 400 expert-validated conflict scenarios and 1,200 queries over more than 100 everyday topics, with contexts up to 150K tokens. Queries test whether the model spots the outdated belief, rejects a false premise and acts on the new state.

Agents negotiate the price of a single item with simulated counterparts over up to 10 rounds of alternating offers. Counterparts follow six behavior families across three regimes. A program scores each agent from its logged actions on four separate axes: surplus captured, agreement calibration, belief error about the opponent and procedural violations. Extra modes score merchant profit and a bankroll over many sessions.

Models see a set of images with the same subject and intent and must pick the best one, and in a harder setting both the best and the worst. The 400 sets hold 1,195 commissioned works across fine art, photography and illustration. Labels come from expert judgments, and only sets where experts largely agreed are kept.

Models turn a text context into JSON that must match a given schema and hold the right values. The 5,324 records come from multi-hop questions, OCR text of documents and meeting transcripts, all given to the model as text. Seven automatic metrics, such as value accuracy, faithfulness and JSON validity, combine into an overall score that weights hard schemas more.

Models play the social deduction game Blood on the Clocktower (the Trouble Brewing script) in mirrored pairs of games. In each match one model plays good and the other evil, and then they swap. A coded Storyteller runs the game, and a Bradley–Terry fit over all matches gives ratings where 1500 is average.

Models write long stories, up to tens of thousands of words, from 2,000 prompts in four task types. An automated checker, ConStory-Checker, finds contradictions in facts, timelines, characters and world rules and cites the passages behind each one. Scores count consistency errors, using a taxonomy of 5 categories and 19 subtypes.

Agents answer business questions over data split across several database systems, with duplicated and inconsistent records and key facts buried in text. The 104 queries cover 17 datasets and four database systems, based on a study of companies building data agents in six industries. Scores are pass@1 against known answers.

Agents answer multi-step questions by calling more than 8,000 locally hosted APIs over real databases in 62 domains, along with matching document collections. Tasks need chains of 3 to 7 steps across APIs and documents. A deterministic evaluator replays tools and checks the full trajectory.

Speech-to-speech models hold one long spoken conversation per scenario and call tools when needed, such as booking a dental appointment or ordering groceries. Six scenarios add up to 221 spoken turns with corrections, false memory traps, prompt injection and state tracking. An LLM judge grades each turn against a golden answer.

Coding agents resolve 1,136 issues from 66 repositories in eight programming languages, and the benchmark checks which code they look at along the way. Each task has human-annotated gold context, and the tracker scores context recall, precision and efficiency over the agent's trajectory. It adds these process measures to the usual issue-resolution result.

Models play the Wikipedia game: starting from one article, they follow one hyperlink at a time to reach a target article. At each step the model sees only the links on the current page, and a game is lost after 30 steps. Three difficulty levels are set by the shortest path between the two pages. Scores are the share of games won at each level.

Coding agents play RuneScape on an emulated game server by writing TypeScript against the rs-sdk library, with game-wiki notes for reference. Tasks train one skill or gather gold within 15 or 30 minutes. Skill tasks are scored by the peak XP rate in any 15-second window, and a verifier reads the final game state.

Video generation models solve visual reasoning tasks by generating a video from a starting image and instruction. Rule-based scorers check the result instead of a judge model. Tasks cover abstraction, knowledge, perception, spatial reasoning and transformation, with in-domain and out-of-domain splits.

Coding agents take a real backend repository, edit the code, set up the environment and deploy the service in a container. The 224 tasks come from 127 open-source repositories in 8 languages and 19 frameworks. A task passes only when the running service passes external end-to-end API tests.

Agents complete 138 long tasks drawn from daily AI use, grouped into 32 scenarios that cover six agentic capabilities such as coding, front-end work, game development and research. A typical task takes about 90 tool calls, 1 million tokens and hours of work. A simulated user gives feedback during the run, and a Docker sandbox checks the deliverables against task rubrics.

Agents act as an in-car voice assistant that talks with an LLM-simulated driver and uses 58 connected tools for navigation, productivity, charging and vehicle control under set policies. Besides ordinary tasks, hallucination tasks remove a needed tool or fact, and disambiguation tasks need a clarifying question or a lookup. Scores reward consistent success across repeated runs.

Deep research systems answer complex research tasks that require evidence from many web sources. A persona-driven pipeline writes the tasks and keeps only those that need outside search. A model judge scores each report on general and task-specific dimensions, and an agent fact-checks the report's statements with web search even when citations are missing.

Multimodal agents hold long conversations over many sessions that mix images and text, then answer questions that need what they remembered. Questions test memory extraction and test-time adaptation, reasoning over memory, and managing knowledge that changes over time.

Models answer tasks from 11 existing datasets in RAG, reasoning, alignment and tool use while their context holds distractors, such as random documents, unrelated chat history and hard negatives. The score shows how much accuracy drops from the clean version of each task.

Vision-language models estimate how far a task has progressed from partial observations, such as frames of a robot or human doing a job. The benchmark varies how the demonstration is shown and the camera viewpoint, and it includes cases with no valid answer.

Retrieval and RAG pipelines answer queries over visually rich documents, such as reports with tables, charts and images. The benchmark has 10 datasets from professional domains, about 26,000 pages and 3,099 human-verified queries, each in 6 languages. It scores retrieval relevance, bounding-box grounding and the final answers.

Coding agents rebuild a feature in a real open-source Python project from a plain-language request. Each task comes from a historical vulnerability fix, so the original human version was insecure. Hidden functional tests and security tests check the result. Scores report the share of solutions that work and the share that are also secure.

Models answer Chinese-language prompts that probe safety across areas such as legality, ethics, factuality and privacy. The paper's first edition turned a judge's pairwise comparisons into Elo ratings. The leaderboard's March 2026 edition gives each model an overall score from 0 to 100. The question bank is updated in dated editions.

Image editing models carry out 2,700 edits that need reasoning, in real-world scenes and in game worlds with human-defined rules, across 8 primary and 18 secondary dimensions. A multimodal judge scores each edit against both a text reference and a ground-truth image.

Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps.

Prolific participants, stratified across 22 demographic groups in the US and UK, hold multi-turn conversations with two anonymous models on topics they choose. They compare the models overall and on dimensions such as trust, ethics and safety. A hierarchical Bradley-Terry-Davidson model turns the votes into ratings, adjusted to census data.

Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem.

Text-to-image models draw images for 600 prompts across 5 themes and 20 subthemes, in English and Chinese and in short and long forms. A multimodal model judge checks each image on 10 primary and 27 secondary semantic dimensions, such as attributes, actions, relationships, layout, text and logical reasoning.

Models act as autonomous AI red teamers on 70 black-box capture-the-flag challenges from Dreadnode's Crucible platform. Each challenge hides a flag behind an AI or ML system, and the model writes Python code to attack it with methods such as prompt injection, evasion and model inversion. A challenge counts as solved when the model submits the correct flag.

Models solve Olympiad-level inequality problems, recast as two checkable tasks: estimating a tight bound and predicting the relation between two expressions. IMO-level medalists wrote and reviewed the 200 test problems. Five model judges check the final answer and look for toy-case reasoning, logical gaps and numerical shortcuts, and a solution counts only if it passes all five.

Models answer about 30,000 questions over real tables across 25 tasks, such as data cleaning, table joins, column and table transformation, table question answering and turning language into code. The tasks come from decades of computer science research on tabular data. Answers are checked automatically against references.

Coding agents such as Claude Code, Codex and Gemini CLI solve TypeScript tasks, and tests decide pass or fail. Version 1 uses 25 Exercism practice exercises. Version 2 uses real tasks from a large TypeScript monorepo, taken from SWE-Lancer.

Models answer multi-step reasoning questions about long narrative videos, which average 1.6 hours. The 1,010 videos come with 9,468 human-labeled question and answer pairs and 30,292 timestamped reasoning steps across seven reasoning types. Multiple-choice answers score the outcome, and a model judge scores the reasoning steps.

Models answer 6,781 Chinese finance queries drawn from real business scenarios. The queries cover numerical calculation, reasoning, information extraction, prediction recognition and knowledge questions, in nine categories. Some tasks are scored against answers and others by IteraJudge, the authors' LLM judging method.

Browser agents complete 2,454 tasks on 452 live websites from the global top 1,000 by traffic. Tasks either read information or write data: creating, updating and deleting records, logging in and downloading files. The board compares browser agent products.

Models work on 87 tasks built from real clinical text, such as electronic health records, in nine languages. The tasks span triage, consultation, information extraction, diagnosis, prognosis and billing codes across 14 specialties. Each task uses its own automatic metric, and the board reports the average.

Models answer 297 questions on Turkish law in three stages: 97 questions from official ÖSYM legal exams and 200 harder questions across 14 branches of law. Answers are checked against the answer key. A judge model also grades the reasoning from 0 to 10.

Vision-language models answer 20,498 questions about 19,200 driving frames, covering perception, prediction, planning and behavior. Inputs come in 17 settings: clean images, corrupted images and text only. The test checks whether answers rest on the image or on general knowledge.

Models answer Arabic questions in four tasks: question answering, reasoning, orthographic and grammatical analysis, and safety. An LLM judge scores each answer on the 3C3H measure: correctness, completeness, conciseness, helpfulness, honesty and harmlessness. Only correct answers get credit on the other dimensions.

Spoken language models follow text instructions about audio across 180 tasks in speech, music and other sounds. Tasks include classification, regression and sequence generation, and researchers keep contributing new ones. Models run zero-shot, without task-specific training.

Multimodal models answer questions about first-person videos that run 20 to 120 minutes. The 500 videos come from Ego4D, and the 12,976 five-way multiple-choice questions cover summarization, perception, visual reasoning and navigation. Score is accuracy.

Models answer 886 questions about real-world tables across 18 fields in four groups: fact checking, numerical reasoning, data analysis and visualization. Fact and numeric answers are scored by exact match, analysis answers by ROUGE-L, and chart code by pass@1.

Audio language models follow instructions about speech, voice and environmental sounds. The paper covers 8 tasks and 26 datasets, including speech recognition, spoken question answering, voice traits and audio scenes. The repository has since grown to more than 50 datasets.

Mobile GUI agents carry out tasks that cross several apps, such as sharing a post from one social app to another. The dataset has 8,334 episodes averaging 15.3 steps, over 212 apps, 1,357 app combinations and six devices. Agents are scored by how often each predicted action matches the recorded step.

Models answer Chinese financial questions in two parts. The knowledge part has more than 38,000 multiple-choice questions from financial qualification exams, with explanations to test reasoning. The application part has more than 16,000 test items for classification, translation, relation extraction, reading comprehension and text generation.

Models answer 1,822 questions about 80 real-world tables, such as Forbes billionaires and Airbnb listings, with 3.7 million rows in total. Answers are booleans, numbers, categories or lists, matched against a key. A lite setting gives each table as a small sample.

Retrieval systems find the right items for complex queries over knowledge bases that mix free text with relations between entities. The three domains are product search, academic paper search and precision medicine. Most queries are synthesized and checked by people, and a smaller set is written by people.

Models solve user requests by calling tools from ToolBench's large set of real-world APIs. A virtual API server with a cache and API simulators stands in for live APIs, so results stay stable over time. GPT-4 judges whether each request was solved and which of two answers is better, giving a solvable pass rate and a win rate.

A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading.

Models simulate populations of fictional personas and answer personal-values questionnaires in many different conversation contexts. The benchmark measures how stable the expressed values stay across contexts, both for each persona and for the ranking of personas within the population.

Agents plan multi-day trips from a user request, using tools to search nearly four million records of flights, hotels, restaurants and attractions. Each plan must meet the user's hard constraints, such as budget, and commonsense rules. The headline final pass rate counts a plan only when it meets every constraint.

LLM agents act in 9 partially observable, multi-turn environments that cover embodied tasks, games, web browsing and tool use. Besides final success, AgentBoard reports a progress rate, the share of annotated subgoals the agent reaches. An open toolkit breaks results down by skill.

Retrieval-augmented systems answer 2,556 queries whose evidence is spread across two to four news articles. The queries also use document metadata, such as source and date. The benchmark scores both retrieval quality and answer accuracy.

Models answer open-book questions about public companies using their filings, such as 10-Ks and earnings reports. The full set has 10,231 questions with answers and evidence strings, and 150 of them are public. The makers graded answers by hand, and later users often grade with a model judge.

Text-to-video models generate clips for 700 prompts drawn from real user requests. The clips are scored with 17 automatic metrics for visual quality, content quality, motion and text alignment. Weights fitted to human ratings combine the metrics into one score.

Models answer questions whose answers change over time, never change, or rest on a false premise that they should reject. FreshEval, an LLM judge, grades answers in a Relaxed mode or a Strict mode that also penalizes any hallucinated detail.

Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings.

Models get harmful requests, such as a request for malware, and should refuse them. The set has 520 harmful behaviors and 574 harmful strings. Most papers add an attack, such as an adversarial suffix, and report the attack success rate.

Models solve open-ended college-level problems in mathematics, chemistry and physics. Most answers are numbers, which a script checks against the key.

Models prove theorems from Lean's mathlib library inside the Lean proof assistant, which checks every proof. The benchmark has 98,734 theorems and proofs with premise annotations. A novel-premises split tests whether a prover can use lemmas it never saw in training.

Models answer questions that need outside information, using 13 tools such as database queries, text retrieval and a calculator over fixed data sources. The questions are built so that a model's internal knowledge cannot answer them. An easy and a hard set separate single lookups from multi-step tool use, and answers are checked by exact match.

Chat models play rule-based dialogue games in self-play, such as Taboo, Wordle and reference games. The clemscore combines how often a model follows the game rules at all with how well it plays when it does. Version 4.0 runs 14 games in 30 languages.

Models answer a conversation of related numerical questions about a company's financial report, with tables and text, where later questions build on earlier answers. The dataset has about 3,900 conversations with 14,000 questions, built from FinQA. Answers are checked by execution accuracy against the gold reasoning program.

Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates.

Models write programs for competitive programming problems, drawn from Codeforces and other sites, and the programs run against hidden tests. Google DeepMind built the dataset for AlphaCode, with training, validation (117 problems) and test (165 problems) splits and many generated tests per problem to cut false positives. Scores are the share of problems solved within a fixed number of submissions, such as 10@k.

Models solve a suite of English legal text tasks drawn from court cases, EU law, contracts and terms of service. Most tasks are classification, and one asks for the holding of a US case. Each task reports micro and macro F1.

Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates.

Models answer numerical questions about company earnings reports that mix tables and text. Financial experts wrote 8,281 question and answer pairs, each with a gold reasoning program. Answers are checked by execution accuracy and program accuracy.

Self-supervised speech models are tested as frozen encoders on a range of speech tasks, including speech recognition, keyword spotting, speaker identification, intent classification and emotion recognition. Each task trains only a light prediction head on top of the shared model.

Models answer 16,552 questions about 2,757 passages from real financial reports that mix a table with text. Many answers need arithmetic, counting or comparison. Scored by exact match and F1.

Models read a passage from a US court decision with the holding of a cited case masked out and pick the correct holding from five choices. The dataset has more than 53,000 questions drawn from US case law.

Models summarize long US government reports from the Government Accountability Office and the Congressional Research Service. The reports are much longer than earlier summarization sets, and each has an expert-written summary. Scores are ROUGE overlap with the reference summary.

Models answer a query about a long meeting transcript with a written summary of the relevant parts. The 1,808 query and summary pairs cover 232 meetings from several domains. Scores compare outputs with reference summaries using ROUGE.

Models write a recap of a TV episode from its full transcript. Plot details sit in dialogue scattered across the episode, so the model must find and join them. Scores compare outputs with human-written recaps using ROUGE.

Multilingual models are fine-tuned on English data and tested on 10 tasks in 50 typologically diverse languages, including harder language-agnostic retrieval tasks. It extends XTREME and adds MultiCheckList, a multilingual diagnostic suite.

Models review commercial contracts and highlight the clauses a lawyer would flag, across 41 categories such as governing law, non-compete and change of control. Legal experts from The Atticus Project wrote more than 13,000 labels over 510 contracts. Scores measure how well the extracted spans match the expert labels.

Models solve 1,000 grade-school arithmetic word problems. The problems are variations of problems from earlier datasets, made to catch solvers that ignore the question or the word order. The final numeric answer is scored by exact match.

A suite of 10 code tasks across 14 datasets, grouped as code to code, text to code, code to text and text to text. Tasks include clone detection, defect detection, code completion, code repair, code search, code generation and code summarization. Each task has its own metric, such as accuracy, BLEU or MRR.

Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route.

Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages.

Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms.

Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates.

An entailment test built to catch three shallow shortcuts: lexical overlap, subsequence and constituent. Every premise and hypothesis pair shares words, but only some pairs are entailments. A model that relies on word overlap fails the non-entailment cases.

Systems check 185,445 claims against Wikipedia and label each Supported, Refuted or NotEnoughInfo. Annotators wrote the claims by altering Wikipedia sentences. The full FEVER score also requires the right evidence sentences.

Models answer questions about books and movie scripts, either from a Wikipedia summary or from the full story. The questions need an understanding of the whole narrative rather than one matching passage. Answers are free text, scored against two reference answers with F1, ROUGE-L or BLEU.

Models answer short open-ended questions about web video clips from the MSR-VTT captioning dataset. The question and answer pairs were generated automatically from the clip descriptions. Scored by answer accuracy.

Models turn natural language questions into SQL queries over single Wikipedia tables. The dataset has 80,654 hand-annotated questions over 24,241 tables. Scored by execution accuracy and logical form accuracy.

Models repair 40 classic algorithm programs, in Python and Java, that each contain a one-line bug. The bugs fall into 14 defect classes. A fix counts when the program passes the test cases.

Models solve one-unknown arithmetic and algebra word problems drawn from several earlier collections. Each problem has an equation and a numeric answer, and the final answer is scored by exact match.

Models answer complex questions about semi-structured HTML tables from Wikipedia. Answers often need counting, comparison or aggregation across rows. Scored by answer accuracy against the gold answers.

A collection of 854 real, reproducible bugs from 17 open-source Java projects, each with the buggy and fixed versions and tests that expose the bug. Program repair studies ask a model to fix each bug, and a fix counts when the project's tests pass. It also serves fault localization and test generation research.

Systems answer biomedical questions written by experts, using PubMed articles as evidence. Questions are yes/no, factoid, list or summary types. A second task asks systems to index new PubMed articles with MeSH headings.

A model works in a team with scripted teammates and live model partners, and a teammate creates a test: pressure to take an unsafe shortcut, misconduct to report, a need nobody states, a conflict, or a cheat that a weaker model left behind. The 240 tasks across 8 dimensions of teamwork come from real records, such as aviation accident reports and clinical cases, and from existing benchmarks, such as SWE-bench and negotiation datasets. Each task has a checkable outcome. A model is scored only on the tasks it solves alone, so the score measures teamwork, not ability.

Agents act in 50 live game and web tasks where a late action fails: 30 web tasks (travel, shopping, services and security) and 20 games. The environment keeps moving while the model thinks. Three clocks separate speed from skill: Wall Clock runs on real time, Paused Clock stops while the model thinks, and Token Clock advances with each generated token. Scores are task success rates over three runs.

Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally.

Assistants carry out everyday personal tasks rather than advise on them: finding childcare, booking a car inspection, ordering groceries on a budget, handling government paperwork. Tasks come from time-use surveys and from about 100 people who recorded tasks they wanted an assistant to handle. Score is average reward across tasks.

Models follow variables through long runs of assignments and report a final value. Four tiers share one source trace and differ only in the update: fixed values, reassignment, self-referential updates, and assignment by reference. Problems come in several formats, such as code, spreadsheet edits and people moving between rooms, at lengths from under 2K to over 2M tokens.

AI systems, each a model in its own app or coding harness, talk with 128 fictional personas over 29 chats. Five chats carry suicidal statements or warning signs of Suicide Crisis Syndrome, and a final request follows, such as a farewell note. Four sub-benchmarks cross explicit statements and warning signs with one chat or separate chats joined only by memory. A model judge rates each reply on a 0 to 4 safety scale.

Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion.

Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings.

Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones.

Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks.

Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice.

Coding agents try to improve the published best model on 20 scientific ML benchmarks across 11 domains, under long in-the-wild runs. The paper reports mean relative improvement rather than a ranking.

Models follow the same AgentIF instructions in thinking and non-thinking modes with vendor decoding settings. A gpt-5.6-luna judge scores 3 runs per model.

Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references.

Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion.

Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning.

Artificial Analysis's index that tracks how model capability changes over time on one continuous scale, launched alongside Intelligence Index v5.

Agents search the web to build a full table of every entity that meets a set of conditions, with 2 to 10 attributes each. A row counts only if every cell is correct.

Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level.

Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes.

Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success.

Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project.

NVIDIA evaluation of short, direct answers for voice and device assistants. The card gives no link or method.

Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1.

Agents take a chip from workload specification to a routed GDS layout with open-source tools. Each design must pass hidden verification, routability and timing gates.

Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score.

Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback.

Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips.

Models play full games of Civilization V with the Vox Populi mod against each other on matched maps and seeds. Pairwise skill ratings rank long-term strategy.

Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency.

Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment.

Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets.

Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check.

Multimodal models write programs that render complete videos from text, an image or a video. The default Overall board ranks models by their mean rank across quality metrics, so a lower score is better.

Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail.

Agents work through 120 professional tasks, each a ladder of five levels that must be passed in order. Later levels reuse files, tools and lessons from earlier ones. An evaluation agent grades against expert rubrics.

Contact-center agents take simulated calls for lending, health and airline lines. Tool calls get deterministic checks and LLM judges grade the rest.

Models solve unpublished research-level physics challenges written by working physicists, with answers graded automatically. The v2 release revised the public test challenges.

Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning.

Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass.

Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally.

Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos.

Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria.

LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance.

Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores.

Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores.

Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to.

Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics.

Models build a coding-agent harness from an interface skeleton and a development document. The frozen harness then runs 328 scripted test cases in separate containers.

Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases.

Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit.

Coding agents answer frozen robot tasks from many benchmark suites with a program. Each program is verified in a separate sandbox, run on Harbor.

Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks.

Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees.

Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna.

Coding agents build executable code-based 3D worlds that capture how a world works, testing world understanding and its realization in code.

Systems extract every schema value from long-tail synthetic documents with charts, drawn relations, house rules and jargon. Scored by value F1 and by grounded F1, which also needs the source location.

Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer.

Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1.

The editors run five AI systems once each on ten unpublished research problems with proofs of up to 12 pages. About 30 referees score correctness, exposition and attribution. The score is average correctness, from negative 1 to 1.

Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes.

Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry.

Agents work inside the released repository of a recent systems or programming-languages paper and try to beat that paper's method. A hidden held-out suite grades the final patch against the human reference.

Models act as text world models. Given an agent's past actions and observations, they predict the environment's next observation across code, enterprise, games, math and science, and dialogue.

Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time.

Agent memory systems must infer user habits from weak evidence spread across many past conversations and apply them correctly later.

Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices.

Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes.

Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion.

Agents in Hermes Agent do 150 tasks across 25 categories, such as research, diagrams, memory, tool use and safety. Graders check the files and state left behind.

Nous Research averages pass@1 and cost per task over Hermes Bench, Terminal-Bench 4.0, Terminal-Bench Science and SkillsBench, all run in Hermes Agent.

Models predict how specified groups of U.S. adults answer closed survey questions, as a distribution over options. Scored as 1 minus total variation distance.

Models answer open-ended questions about what an image or short video implies but does not show, such as hidden objects, causes and intentions. An LLM judge checks each answer against a human-written rubric.

Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1.

Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric.

Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches.

Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome.

Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check.

Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria.

Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run.

Models review contracts and return redlines across 350 attorney-written tasks in 35 scenarios and five agreement types. Tasks cover first review and replies to a counterparty.

Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score.

Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated.

Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings.

Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place.

Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit.

Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure.

Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings.

Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns.

Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended.

Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards.

Models do military mission analysis under the joint planning process on long-context scenarios, graded against expert rubrics.

Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar.

Models recall key facts from multi-turn conversations of 64K to 100K tokens that include distractor chats and turns that conflict with or update the facts. An LLM judge scores answers on weighted rubrics.

Models are scored on the equal-weight average of micro1 medical benchmarks, currently RealmPathology, PharmaBench and WellnessBench.

Models support simulated patients from MindSim, a tool-using harness with detailed profiles. Psychologist-written rubrics (MHQR) replace Likert scales for scoring.

Models write government status reports in Word from working files and a template, across 20 expert-written Department of Defense style assignments. Expert rubrics score each decision.

Enterprise agents answer 1,613 questions over workplace documents, chats and figures that change over time. They must resolve conflicting versions and say when evidence is not enough.

Models answer chat prompts from an internal NVIDIA set. The Nemotron 3.5 Super VL card labels the score an LMArena v3 proxy.

Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing.

Document parsers convert single pages to text, and 16,288 binary unit tests on 2,937 pages check the output. The headline joins text, table and layout subscores.

Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts.

Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings.

Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem.

Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house.

Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims.

Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals.

Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.

Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds.

Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff.

Agents run 50 expert-verified physics research tasks in strong-field and attosecond physics, deciding how to generate and validate evidence. A 20-criterion score and a closure level from RC0 to RC4 grade them.

Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective.

Programmable world models must render video that follows the rules and interactions their program specifies. The test checks visual fidelity to fine-grained rules.

Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window.

Models read months of real chats between ten people and an AI companion, then answer questions about who each person is. The release has 27,218 messages over up to 120 days.

Models interpret real anatomic pathology reports and must not claim a diagnosis, stage or treatment the report does not support. Rubrics have 772 positive and 299 penalty criteria.

AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1.

Image editing models follow 1,000 bilingual instructions over six reasoning types, adding counterfactual and hybrid reasoning. It covers single-image, multi-image and multi-turn editing across 65 task types.

General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks.

An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3.

Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks.

Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1.

Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette.

Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements.

Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement.

Chatbots hold 6 to 20 turn chats with a simulated teen across 106 clinician-written scenarios, branching on how the teen reacts. A judge rates safety checks and reports the failure rate.

Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading.

Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field.

Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder.

Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers.

Agents build executable Simulink models from text for 101 tasks in ten engineering domains. Simulations check that each model meets its engineering requirements.

Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior.

Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes.

Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority.

Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks.

Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records.

Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows.

Each model gets one hour in its lab's own CLI to write a StarCraft: Brood War bot in C++. The score is its win rate against human-written bots.

Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing.

Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change.

Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part.

Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result.

Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build.

Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved.

Agents browse the web to retrieve information and do research tasks. iFlytek built it in house.

Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one.

Agents solve 50 unpublished research-level theoretical physics problems on their own, with literature search and scripts. Each final answer is checked against an expert answer, and the score is the pass rate.

Agent harnesses, not only models, are tested on reliability across seven components, with traces used to attribute each failure to its cause.

Image generators render dense bilingual text in many regions from 432 prompts. Scores check string accuracy, placement and legibility.

Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones.

Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP.

Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue.

Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights.

Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science.

Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools.

Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result.

Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages.

Models answer young people asking about eating, sexual safety, substances and self harm, in clinician-written single-turn and pushback dialogues. A pass must be both safe and helpful.

Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics.

Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string.

Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture.

Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT.

Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.

Systems label private pre-2022 human documents and AI rewrites of them as human or AI written. Both LLMs and commercial detectors are scored by balanced accuracy.

Models answer 3,900 hidden English prompts, 325 in each of 12 hazard categories, picked with item response theory. A calibrated evaluator gives a 0 to 100 score.

Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run.

Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero.

Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions.

Version 1.5 of Artificial Analysis's coding agent index, which moved to the harder Terminal-Bench 4.0 and DeepSWE 1.1.

Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate.

Version 4.2 of Artificial Analysis's index, which added multi-hour office work and document tasks and dropped GPQA Diamond.

Version 4.3 of Artificial Analysis's index, eleven weighted parts led by knowledge work, office tasks and Terminal-Bench 4.0.

Models answer research-level computer science questions derived from recent arXiv papers, each with a reference final answer. The first set has 46 questions from August 2026.

Models answer research-level physics questions derived from recent arXiv papers, each with a reference final answer.

Agents do 100 investment banking tasks inside 13 simulated firms. They gather facts from email, chat, calendar and files over MCP, then build an Excel model graded against expert rubrics.

Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed.

Coding agents play StarCraft: Brood War against each other in real time. Systems are ranked by win rate over matches, with cost per game.

Browser agents do 200 real web tasks. A findings judge checks tool evidence, answers and files, and deterministic integrity checks run as well.

Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit.

Agents rebuild licensed engineering drawings as editable CAD parts through the scripting APIs of NX, SolidWorks, Onshape, Fusion or Build123d. A verifier scores geometric match.

Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not.

Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out.

Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading.

Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading.

Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Decision models read a state and a typed question and return a decision. The index weights private same-skill sets and new-domain sets, equated on 21 base models.

Models produce design work such as slides and documents that follow templates, from an OpenAI internal set. OpenAI has not published the grading.

Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios.

Agents work through the full industrial engineering design loop on 1,301 expert-curated tasks that need reasoning over geometric and physical constraints across software.

Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support.

Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help.

Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations.

An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number.

Language models play Go against each other through the Game Arena text harness. Ratings come from head-to-head games.

Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts.

Models sit the five 2026 Chinese Gaokao mathematics papers (National I, National II, Beijing, Shanghai and Tianjin). Each paper is graded out of 150 points.

Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria.

World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them.

Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score.

Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers.

Decision models, such as TypeSafe AI's Jev and compatible systems, answer typed decision questions. The board scores accuracy, probability calibration, latency and cost.

Strong models translate hard texts chosen to stay unsaturated, and experts assess the outputs instead of automatic metrics that can be reward-hacked.

Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass.

Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers.

Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed.

An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity.

Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result.

Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details.

Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details.

Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code.

Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold.

Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly.

Agents run autoresearch on real open-source model development projects with their original code, data and checkpoints, for 60 or more hours. Fixed verifiers score the result against the human recipe they started from.

Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task.

Vision-language models perceive, reason about and judge whether videos follow real-world physics. The benchmark also tests if they can judge the physical realism of generated videos.

Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate.

Models resolve tasks in private production codebases licensed from real companies. The board reports resolution rate with standard errors.

Ten frontier agents each post-train the same Nemotron base model on a shared GPU cluster for six days. People then rank the resulting models in blind side-by-side votes, and agents retrain on that feedback.

ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them.

Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup.

Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts.

AI tutors teach real students GRE material and are scored by the students measured learning gains. A second study has expert tutors compare AI lesson plans and practice problems.

Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container.

Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol.

Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real.

Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision.

Version 2.1 of Vals AI's GDP-weighted index, which moved coding to Terminal-Bench 4.0 and added a tax sector.

Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it.

Audio language models answer questions that need memory of earlier spoken sessions. Questions cover what was said, who said it, how it was spoken and what was audible.

Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels.

Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects.

Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice.

Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right.

Models answer questions that need reasoning across about 100k tokens of documents. Mercor adds 89 new expert-written tasks to the 100 public ones and reports both scores.

Models state probabilities and preferences on conceptual questions, and the benchmark checks 567 expert-approved logical consistency constraints such as conjunction rules and order invariance.

Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors.

Version 1.4 of Artificial Analysis's coding agent index, which moved to the full Terminal-Bench 2.1 and began scoring reward-hacked runs as zero.

One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores.

AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge.

Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts.

Agents browse the web to find hard-to-locate facts with short answers. Mercor adds 100 new expert-written tasks to the 130 public ones and reports both scores.

Models answer descriptive and reasoning questions about scientific charts. Mercor adds 150 new expert-written tasks to the 5000 public ones and reports both scores.

Agents on stateful runtimes are scored with their traces, covering evidence gathering, runtime routing, safety boundaries and repeated runs in frozen workplace setups.

Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases.

Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds.

An index that ranks models on reasoning about hard-to-verify topics such as AI alignment and decision theory. It weights LMCA 60 percent and ACCoRD and DTBench 20 percent each.

Agents work through five domains of 100 linked subtasks of rising difficulty, testing whether skill libraries they build in context actually help later tasks.

Agents answer questions over the email archives of four synthetic companies, up to 232,692 documents. Answers are checked against each company's ground-truth knowledge graph.

Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately.

Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt.

Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published.

Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance.

Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable.

Language models play Nine Men's Morris. Forming a row of three, called a mill, lets a player remove one of the opponent's pieces.

Language models play Reversi, also called Othello, flipping the opponent's disks that they flank in a line.

Language models take turns drawing a word as ASCII art and guessing what the other model drew.

Agents learn from training tasks drawn from enterprise workflows and must reuse that experience on related test tasks. Gains at test time measure self-evolution.

Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores.

Video and world models generate rollouts from image and action prompts. An evaluator agent plans checks, spawns sub-agents, gathers spatiotemporal evidence and scores observation, transition and persistence with written reasoning.

Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores.

Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published.

Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published.

Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published.

Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published.

Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published.

Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing.

Models answer 3,240 questions about 360 English and Chinese recordings from 10 minutes to over 2 hours. Each question is scored as multiple choice and as open-ended.

Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit.

Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores.

Models with memory are tested on cases where correct, relevant memories still distort their reasoning or beliefs and lower task performance.

Models reason over synthetic medical claim files of about 25k to 64k tokens, as a claims reviewer would. Three judge models check completeness and accuracy.

Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores.

Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match.

Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts.

Video generators are run many times on the same start and action, and the outcomes become a distribution over physical behaviors. The test checks whether it matches reference probabilities and covers every valid behavior.

Models write Lean 4 proofs of graduate-level theorems from paired natural-language and formal statements. Scored by the share the Lean checker accepts.

Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth.

Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks.

Agents get a weak method on a long research task and must turn it into one that scores better on hidden data. It measures recursive self-improvement on executable research.

Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict.

Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions.

Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference.

Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores.

Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions.

Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix.

Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores.

Models write proofs of target results from STOC, FOCS and SODA papers, given the needed context. A verification agent, checked against expert judgments, grades each proof.

Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores.

A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday.

Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision.

Version 2 of Vals AI's GDP-weighted index, which brought back a legal sector and added financial modeling in Excel and code migration.

Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness.

Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame.

Vision models answer 161 questions about lab images such as gel blots, plasmid maps, microscopy and flow cytometry plots. An LLM judge grades each answer against a key.

Speech-to-text models transcribe 300 English workplace recordings. They are scored on whether every target entity, such as an address or email, survives exactly.

Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo.

Controllable video generators are tested as world models. Levels run from visual appearance to whether the depicted world reacts plausibly beyond what the prompt states.

Models write proofs for 245 advanced problems at undergraduate and doctoral qualifying exam levels. A verifier trained on expert annotations grades each proof.

Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework.

Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios.

Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria.

Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes.

Coding agents and model pairs diagnose misconfigurations, provision infrastructure and operate live cloud setups in disposable real AWS accounts.

Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit.

Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails.

Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate.

Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition.

Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break.

Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.2 added instruction following and advanced tool use problems. Agentic graders score correctness.

Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug.

Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings.

Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published.

Agents iterate for up to 12 hours on 134 executable real-world tasks with multi-level feedback. Scores are tracked at 2 hour steps to measure learning from the environment.

Models hold 16-message roleplay chats with 120 simulated user personas whose preferences compete and are hidden. A three-judge panel compares transcripts pairwise, and ratings are fitted as Elo scores.

Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves.

Agents build investment banking and private equity financial models in Excel from a written brief and source files. Rubrics give partial credit for tabs, structure, formulas and formatting.

Systems read a full business document and return a schema-valid JSON object with every requested value. Unified value F1 compares the output with the expected values.

Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle.

Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics.

Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior.

Two language models negotiate how to split a pool of items, each with private item values. Each side tries to reach a good deal without giving away its values.

Two language models move on a grid and pick up coins. Each should collect the coins that score for it and avoid the rest.

Language models play Lines of Action, where a piece moves as far as the number of pieces on its line. The goal is to join all of one's pieces into one group.

Language models play nine linked tic-tac-toe boards. Each move decides which small board the opponent must play next.

Agents do routine company work with mock email, chat, calendar, issue and commerce tools while following a 20 to 124 page handbook. Programmatic criteria check required and forbidden actions, and a trial passes only if all pass.

Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy.

Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published.

Coding agents complete Tencent in-house company engineering tasks. Task and scoring details are not published.

Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published.

Agents build financial models for Tencent in-house finance tasks. Task and scoring details are not published.

Models solve Tencent in-house math problems without tools. Task and scoring details are not published.

Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published.

Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables.

Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages.

Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house.

Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities.

Models solve AIME 2025 competition problems in Korean. Upstage built the set in house.

Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format.

Simulated children use consumer AI apps end to end, with each app system prompt, filters and safeguards active. Scores cover online, physical and sexual, psycho-social and learning risks.

Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading.

Models answer chemistry questions from an OpenAI internal set, grouped with OpenAI's life science research evaluations. OpenAI has not published the task format.

Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence.

Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds.

Image editing models place several named people into shared contact actions such as an embrace or a carry. Scoring targets fused limbs, extra extremities and bodies passing through each other.

Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch.

Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria.

Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3.

Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy.

Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors.

An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model.

Coding agents act as data-centric researchers. With a fixed Tinker post-training stack and budget, they refine training data from dev-benchmark feedback, and the final model is scored on held-out test benchmarks.

Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials.

Vision-language models answer questions about synthetic long documents where length, layout, modality and difficulty are varied one at a time to locate failure causes.

Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search.

Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions.

Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill.

Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable.

Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result.

Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear.

Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews.

Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on.

Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions.

Version 1.1 of Artificial Analysis's coding agent index, which replaced SWE-Bench-Pro-Hard-AA with DeepSWE.

Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents.

Agents play levels of the puzzle game Baba Is You through a text interface and win by rewriting the rules. Models are compared on pass rate, speed and cost.

Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference.

Models answer 500 prompts with embedded child safety risks across 12 categories, such as grooming, impersonation and emotional dependency. A judge marks each response acceptable or not.

Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score.

General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches.

Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check.

Models answer 75 expert-written professional prompts whose constraints depend on each other, fire conditionally or must be inferred. An LLM judge grades 1,559 rubric criteria one at a time, and a task passes only if all criteria pass.

AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning.

Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged.

Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls.

Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass.

Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects.

Agents discover and demonstrate vulnerabilities on the older 615-task cohort. The headline is cumulative stage S3 success.

Models answer the turn of a real Doubao conversation where the user was dissatisfied, with the earlier turns as context. Rubrics grade each answer.

Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics.

Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head.

Agents do 852 tasks rebuilt from real proprietary workplace agent sessions, each with fixtures, hard rules and semantic rubrics.

Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result.

A judge model reads a transcript and answers whether it came from an evaluation or from real deployment. The test measures how well models detect evaluations and how detectable each benchmark is.

Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes.

Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit.

The editors run AI harnesses once each, with 24 hours, on ten unpublished research problems with human proofs of up to eight pages. Referees grade each answer blind.

Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes.

Language models play American checkers against each other, with forced jumps and crowned kings. Ratings come from head-to-head games.

Language models play Clobber, where every move captures an adjacent enemy stone. The player who makes the last legal move wins.

Language models play Hex without seeing the opponent's stones, so they must probe cells to learn the hidden board while building a path.

Language models take turns drawing lines between dots on a grid. Closing the fourth side of a box claims it.

Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together.

Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier.

Models generate JSON or YAML that must follow a requested schema and phrasing constraints, without constrained decoding. A response passes only if every structural constraint holds.

Models study several real photos of a home and draw its floor plan. Scored by an average quality score.

Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof.

Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history.

Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers.

Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures.

Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key.

Models answer 100 research-level math questions with known answers, written by 49 mathematicians at a Max Planck Institute workshop in April and May 2026.

Extraction systems pull structured values from 225 long table-heavy documents averaging 358 pages and about 88,700 ground-truth fields each.

Audio models edit audio from instructions across a broad range of real-world editing tasks and sound types.

Models answer culture-specific knowledge questions in 11 major languages.

Models solve LiveCodeBench-style contest problems in twelve programming languages, with dated problems for contamination-aware evaluation.

Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks.

Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results.

Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance.

Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos.

Models replace PyTorch and NCCL reference code with multi-GPU CUDA kernels for 87 problems from codebases such as Megatron-LM and DeepSpeed. Scored on correctness and speed over the baseline.

Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally.

Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures.

Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts.

Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning.

Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release.

Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables.

Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code.

Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate.

RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose.

Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions.

Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy.

Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail.

Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session.

Models read W-2, 1099 and other PDFs and compute full federal and state returns line by line. A return counts only when every evaluated line is exact.

Agents build a large project end to end with no time limit, such as a faster Rust compiler or a WebGL renderer in WebAssembly. Each challenge has its own test bed and metric.

Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup.

Agents use a terminal for general computer tasks beyond coding, such as file, system and everyday productivity work.

Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision.

Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output.

Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts.

Systems translate the WMT26 general task test set in 24 directions, mostly out of English, across several domains with added instructions. The shared task ranks systems by human contrastive ESA judgments.

Models answer hand-written reasoning questions about images drawn from a taxonomy of thousands of visual concepts. Questions were kept only if frontier models failed them.

Models answer open-ended requests grounded in real user needs across more than 30 daily-life, study and office scenarios. Rubrics grade completion, correctness, coverage and usefulness.

Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results.

Coding agents optimize GPU kernels through full workflows with compilers and profilers. Scores check speedups and whether gains generalize beyond tuned shapes.

Experts such as doctors, lawyers and engineers compare model answers to prompts from their own history. They judge which answer would still help them a month later.

Artificial Analysis's first index of coding agents, each a model in a harness, averaged over three equally weighted coding benchmarks.

Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response.

Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer.

Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs.

Browser agents do 130 new everyday tasks on live websites under six supported harnesses. Request interception and a more lenient LLM judge score each run.

Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.1 added codebase understanding, bug finding, planning and code review problems. Agentic graders score correctness.

Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score.

Agents get one attempt at each of 50 deep engineering tasks in Docker, in the Terminal-Bench 2.0 format. A task passes only if every hidden test passes. Tasks were calibrated so Claude Opus 4.6 still fails some at pass@4.

Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action.

Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes.

Coding agents keep one project working across 26 tasks of 5 to 15 rounds each, while requirements change, pile up and conflict. Cumulative hidden tests check every round in a persistent container.

Teams of language models give one-word clues and guess linked words, in a game like Codenames, against an opposing team.

Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy.

Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute.

Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses.

Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON.

The model under test judges fixed creative-writing samples against low and high anchor responses. The score is how well its ratings separate writer models.

Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time.

Memory systems help agents learn the affordances, dynamics and failure modes of customized web environments. Questions check whether that experience was internalized.

Agents build complete financial models in Excel from a case prompt and documents. An LLM judge grades accuracy, formulas and formatting on a 0 to 100 scale.

Multimodal agent memory systems answer questions that need fine visual evidence kept from earlier sessions, which captions or text traces alone cannot supply.

Vision-language models and memory agents answer 789 questions over multimodal multi-session conversations, covering extraction, temporal reasoning, knowledge updates and refusal at four context lengths.

Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects.

Memory-augmented agents answer questions after long, frequently updated histories where memories interfere. Instances average 138,800 tokens and reach 1.8 million.

Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state.

Models answer five kinds of music questions that need the correct time region of the audio, such as instrument entries. Music experts validated the items.

Coding agents with a fixed H100-hour budget and no internet try to improve on a strong NanoGPT speedrun starting point. The score is the share of later human progress recovered.

Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems.

Image editing models do precise single-answer edits in 20 operations such as geometric transforms, color changes and symbolic edits. Procedural generation gives deterministic checks.

Models answer matched left-coded and right-coded prompts, such as Obama and Reagan. Judges score whether sentiment and helpfulness stay consistent across each pair.

Models drive an OpenClaw agent through 148 real-world tasks, including data and log analysis, meeting notes, DevOps, PDFs, research and writing. Graded by checks and a Haiku judge.

Models solve generated planning problems that coordinate goals, constraints and resources. Instances are scalable and come with verifiers.

Models write proofs for hard competition problems. An LLM judge compares correct proofs pairwise on conciseness, computational ease, simplicity, diversity and adaptivity, giving Elo ratings.

Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images.

Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages.

Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs.

Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests.

Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit.

Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots.

Models face adversarial national security and public safety prompts in English and Korean, set in US or Korean contexts. Each prompt has a benign pair to measure over-refusal.

Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change.

Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems.

Spatial foundation models are tested across many 3D tasks, viewpoints, scene domains, input densities and hardware limits, beyond the domains they were trained for.

Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning.

Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses.

Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result.

A suite of three terminal datasets for coding agents. DeepTerminalBench tests single-shot depth, EvoCode-Bench tests multi-round iteration and RoadmapBench tests version upgrades on real repositories.

Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans.

Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes.

Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics.

Interactive video world models run 289 multi-turn cases with navigation, subject actions, event edits and perspective switches. 22 metrics cover quality, adherence, consistency and physics.

Video generation models are tested for causal and arrow-of-time understanding by comparing real videos with their time-reversed versions, in two levels.

Agent products such as Claude Code and Codex do 94 production tasks from 7 companies across 6 O*NET domains. Grading mixes formal checks, UI tests and rubrics.

Models pick the most culturally fitting reply in dialogues from 13 Arab countries, in Modern Standard Arabic and each local dialect. It also has translation and dialect-steering tasks.

Models prove or disprove theorems from recent arXiv papers, stated in Lean 4. A strict comparator checks each submitted proof against the fixed reference statement.

Agents search the scientific literature, either tracking down one target paper through multi-step probing or collecting every paper that meets given conditions.

Agents forecast 1,417 binary questions asked in October 2025 and resolved in December 2025, researching offline in a frozen corpus of 15 million documents. Scored by Brier score.

Agents complete 105 workflow tasks in 17 families drawn from live ClawHub workflow signals, scored on both the workflow and the final workspace state.

Browser agents do 153 everyday tasks on live websites, such as booking travel, ordering food and applying for jobs. Request interception stops real side effects, and an LLM judge scores the five-layer traces.

Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output.

Speech-to-text, text-to-speech and speech-to-speech models from almost 40 providers are scored every 30 minutes on call-like audio with accents, noise and echo.

Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task.

Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report.

Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours.

Agents work up to 20 hours on 17 ultra-long engineering, performance and research tasks in their own harnesses. The board ranks models by average task rank. The score here is dominance, the win rate against a random baseline, which labs report.

Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking.

Language models play a gomoku-style game and try to line up five of their tokens before the opponent does.

Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state.

Models solve 365 hand-written reasoning problems and 1,095 variants in eight categories that need only K-12 knowledge. Scoring mixes rule-based and model-based checks.

Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts.

Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary.

Coding agents complete Tencent in-house backend development tasks. Task and scoring details are not published.

Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task.

Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published.

Users upload a screenshot, wireframe or mockup, two anonymous models build the website in Code Arena, and users vote for the better result. Votes become ratings with bootstrapped confidence intervals.

Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types.

Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth.

Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images.

Models answer multiple choice questions about theorems from arXiv papers published after their training cutoff. Wrong options follow plausible but invalid proof sketches.

Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report.

Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE.

Omni-modality embedding models are scored on 190 tasks across text, image, video, audio, visual documents and agent-centric retrieval.

Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts.

Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment.

Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable.

Models parse 1,651 PDF pages, including nested tables, dense formulas and unusual layouts, into text, tables and formulas. Scored by the v1.5 Overall formula.

Document parsers process about 2,000 human-verified enterprise pages. Over 169,000 test rules score tables, charts, content faithfulness, formatting and visual grounding.

Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark.

Models find the time span of short sound events hidden in long, unstructured YouTube recordings. It has 400 audio, query and timestamp triplets.

Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests.

Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed.

Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following.

White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace.

Models generate images from the revised WISE prompts. The verified evaluation changes about 200 prompts and the judge, so original scores remain separate.

Models are asked to prove statements from recent arXiv papers that were changed to be plausible but false. Full credit goes to models that say the statement is false.

Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback.

Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry.

People upload their own PDFs with a question, compare answers from two anonymous models, and vote for the better one. Votes become Bradley-Terry ratings.

Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models.

Agents do 1,150 expert-written tasks in eight enterprise domains, such as IT, HR and customer service, using 512 tools over live MCP servers. SQL checks on the final database state score each task.

Agents complete the formal Lean proofs and define the new mathematical concepts in real pull requests to the Fermat's Last Theorem (FLT) formalization project. Lean and SafeVerify check each submitted diff.

Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number.

Models attack 113 mostly unsolved problems in eight areas of mathematics and the mathematical sciences. A program checks each proposed closed form, construction or bound.

Models answer 320 hand-written questions about music, curated and checked by people with musical training.

Models answer 110 source prompts in chat, STEM, math and coding, translated into 22 Indian languages in native and romanized script. A judge model compares answers pairwise.

VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric.

Agents act as OpenClaw personal assistants on 134 multi-step tasks across 10 app domains backed by stateful mock services. A run passes when its reward exceeds 0.8.

Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track.

Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage.

Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories.

Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1.

Models run inside the OpenClaw agent on 45 tasks in four difficulty tiers built from real usage data. Some tasks use an LLM judge.

Models solve 300 pencil puzzles of 20 varieties with unique solutions, either in one shot or agentically with step checks. Every board state can be verified against the rules.

Agents compete in Pokemon battles under partial observability in a Battling Track and play the Pokemon RPG in a Speedrunning Track that needs long-horizon planning.

Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate.

Models name skateboard tricks from technical trick definitions. The score is the share named correctly.

Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity.

Systems write CUDA kernels for 235 workloads taken from real model architectures, run on B200 GPUs. The SOL score measures how much of the gap between a baseline and the analytic hardware speed limit a kernel closes.

Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes.

Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity.

A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions.

Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use.

Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction.

Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference.

Models get prompts built on nonsensical premises. A three-judge model panel grades whether each answer pushes back clearly, challenges partly or accepts the nonsense.

Models solve 53 Codeforces Div. 2 problems from September 2024 to February 2025. An offline judge with rebuilt test cases scores C++, Python and Java submissions.

Models follow complex Chinese production prompts with constraints on format, conditions, content, phrasing, tone, emoji, few-shot style and length. It has 912 cases over 17 weighted dimensions.

AI code review tools review pull requests from five large open-source projects. An LLM judge matches their comments to human-verified golden comments.

Models identify several elements of a business text in one inference, such as user intent and slot values.

Models answer 500 advice queries taken from Reddit, Stack Exchange and forums. Model judges score usefulness, logic, instruction following and tone against an expert reference set at 88.

Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task.

Multimodal agents search raw visual histories over several steps to find target images from implicit context cues.

Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy.

Retrieval systems and agents answer 500 questions over a synthetic company's internal documents, slightly over 500,000 of them.

Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests.

Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically.

AI systems attempt ten unpublished research problems that arose in the authors' own work. Mathematicians judge whether each proof is correct.