Coding
We track 317 benchmarks in coding. They run from everyday level to research frontier level.
Anthropic held the top coding score 25% of the contested time
That's about 11 of the 46 months since January 2023, averaged over these benchmarks. We only count time while other labs are still testing, and stop once a model passes 85%. Faded stretches no longer count.
OpenAI Anthropic DeepSeek Google DeepMind Ai2 Reka Upstage Meta Mistral AI Z.ai NVIDIA Sarvam AI Baidu Inception Labs ByteDance Seed Qwen Other labs Meituan MiniMax LG AI Research Xiaomi Corp Ant Group Moonshot AI
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Competition The strongest students, in olympiad-style contests
Expert Specialists with years of training, often over hours of work
Sources: Official leaderboards, Epoch AI runs (CC BY 4.0) and labs’ model cards
Share of contested time at number one
Anthropic 25% OpenAI 18% 
DeepSeek 14% 
ByteDance Seed 8% 
Qwen 7% 
Google DeepMind 6% 
Mistral AI 4% Moonshot AI 4% Inception Labs 3% Z.ai 3% 
NVIDIA 3% 
Meta 2% 
Meituan 1% Other labs 1% Xiaomi Corp 1% 
MiniMax 1%
DeepSeek 14% ByteDance Seed 8% Qwen 7% Google DeepMind 6% Mistral AI 4% Moonshot AI 4% Inception Labs 3% Z.ai 3% NVIDIA 3% Meta 2% Meituan 1% Other labs 1% Xiaomi Corp 1% MiniMax 1% Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
Closed out: 8 benchmarks passed 85%
Anthropic 4 DeepSeek 1 Google DeepMind 1 
Ai2 1 OpenAI 1
DeepSeek 1 Google DeepMind 1 Ai2 1 OpenAI 1 Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
- Sep 2026 SWE-Bench Pro V2 Claude Opus 5
- Jun 2026 Vibe Code Bench Claude Fable 5
- Apr 2026 SWE-bench Multilingual Claude Mythos Preview
- Dec 2025 LiveCodeBench DeepSeek-V3.2-Speciale
- Aug 2025 LiveCodeBench v6 Gemini 2.5 Deep Think
- Nov 2024 HumanEval+ Tulu 3 405B
- Apr 2024 HumanEval GPT-4 Turbo Apr 2024
- Mar 2024 MBPP Claude 3 Opus
Each benchmark counts from the one source its own page leads with: its official leaderboard when that is current, else Epoch AI’s own runs, else labs’ own numbers. We date independent runs by the model’s release and lab reports by when they came out.
EleutherAI OpenAI Mistral AI Anthropic DeepSeek xAI Google DeepMind Meta
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Competition The strongest students, in olympiad-style contests
Expert Specialists with years of training, often over hours of work
Sources: Official leaderboards and Epoch AI runs (CC BY 4.0)
Share of contested time at number one, independent runs only
OpenAI 57% Anthropic 37% Google DeepMind 2% Meta 1% Mistral AI 1% DeepSeek 1%
Google DeepMind 2% Meta 1% Mistral AI 1% DeepSeek 1% Sources: Official leaderboards and Epoch AI (CC BY 4.0)
Every independent run: each benchmark’s own leaderboard, Epoch AI’s runs and boards Epoch AI copied, dated by the model’s release. We leave out benchmarks that only labs report.
EleutherAI OpenAI Anthropic Upstage Ai2 Baidu Meta Mistral AI Z.ai DeepSeek Qwen xAI Google DeepMind Ant Group Moonshot AI Inception Labs ByteDance Seed
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Competition The strongest students, in olympiad-style contests
Expert Specialists with years of training, often over hours of work
Sources: Official leaderboards and labs’ model cards
Share of contested time at number one, claimed scores included
OpenAI 52% Anthropic 27% DeepSeek 4% Google DeepMind 3% ByteDance Seed 3% Qwen 3% Moonshot AI 3% Meta 3% Z.ai 1% Inception Labs 1% Mistral AI 1%
DeepSeek 4% Google DeepMind 3% ByteDance Seed 3% Qwen 3% Moonshot AI 3% Meta 3% Z.ai 1% Inception Labs 1% Mistral AI 1% Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
Closed out: 10 benchmarks passed 85%
Anthropic 5 OpenAI 4 Google DeepMind 1
Google DeepMind 1 Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
- Sep 2026 SWE-Bench Pro Claude Opus 5.5
- Sep 2026 SWE-Bench Pro V2 Claude Opus 5
- Jun 2026 Vibe Code Bench Claude Fable 5
- Apr 2026 SWE-bench Verified Claude Mythos Preview
- Apr 2026 SWE-bench Multilingual Claude Mythos Preview
- Aug 2025 LiveCodeBench v6 Gemini 2.5 Deep Think
- Apr 2025 LiveCodeBench o4-mini
- Apr 2024 HumanEval+ GPT-4 Turbo Apr 2024
- Nov 2023 MBPP GPT-4 Turbo Nov 2023
- Mar 2023 HumanEval GPT-4 Mar 2023
Every score, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.
From everyday to research frontier level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
Everyday
Most adults, without training
- DesignArena Website
- DesignArena Game Dev
- DesignArena Code
- DesignArena UI Component
- DesignArena 3D Design
- DesignArena Data Visualization
- DesignArena SVG
- DesignArena Agentic Overall
- DesignArena React Native
- Image-to-WebDev Arena
- DesignArena Frontend Web App
- DesignArena Image to HTML
- DesignArena Full Stack
- DesignArena Android (Kotlin)
- DesignArena Image to Frontend
- StructEval
- DesignArena Agentic Game Dev
- DesignArena Godot Game Dev
- OpenDesign Arena
- DesignArena Video to Website
- SArena
- Chart Arena
- Landing Page Arena
- BigCodeArena
- Design2Code
- DesignArena Blender
- DesignArena Agentic Image to HTML
- Floorplan Arena
- U-Artifacts
- Copilot Arena
- DesignArena Builder
- DesignArena Mobile Builder
- RepoChat Arena
School
Students, from grade school to high school
University
Undergraduates and entry-level professionals
- BigCodeBench-Hard
- BigCodeBench
- Aider Polyglot
- DAXBench v2
- LiveSQLBench
- AutoCodeBench
- Bird-SQL
- ArtifactsBench
- HumanEval-V
- SEAL Coding
- MHPP
- FullStackBench
- ArtifactsBench (Gemini 2.5 Pro 0506 judge)
- GameDevBench
- ChartMimic
- QwenSVG
- ts-bench v1
- Flame-React-Eval
- NaturalCodeBench
- QwenWebBench
- QwenWebDev
- OctoCodingbench
- QwenBlenderBench
- Spider
- ts-bench
- QwenSVGBench
- QwenVision2Code
- CanItEdit
- APPS
- DS-1000
- KAT Code Bench
- HumanEvalPack
- LeetCode Weekly Contest
- DS-Arena-Code
- DS-FIM-Eval
- LBPP (Less Basic Python Problems)
- SVG-Bench (MiniMax)
- CodeUtilityBench (LG)
- Coding (MiniMax in-house)
- HumanEval-COBOL (Cohere)
- Leetcode (easy)
- Leetcode (hard)
- Leetcode (medium)
- Multi-dialect SQL evaluation (Cohere)
- ClassEval
- DataBench
- Surge AI blind coding evaluation
Competition
The strongest students, in olympiad-style contests
Expert
Specialists with years of training, often over hours of work
- SWE-bench Verified
- SWE-Bench Pro
- SWE-bench Multilingual
- Vibe Code Bench
- DeepSWE
- SciCode
- Code Migration
- APEX-SWE
- NL2Repo-Bench
- SWE Atlas
- SWE-bench Verified Extended
- FrontierCode 2.0
- SWE Atlas Codebase QnA Extended
- AndroidBench
- OpenHands Index
- SWE-bench Multimodal
- FrontierCode 1.1
- Code Migration v1.1
- GSO-Bench
- Multi-SWE-bench
- SWE-Marathon
- ReactBench
- SWE Atlas v1.1
- Vibe Code Bench 1-100
- CursorBench 4.0
- Spider 2.0
- SWE Atlas Test Writing
- Senior SWE-Bench
- Vision2Web
- CursorBench
- SlopCodeBench
- SWE-Together
- SWE-Marathon v1.1
- FrontierCode
- SWE-Lancer
- Brokk Power Ranking
- sudo L7
- SWE Atlas Refactoring
- SWE-fficiency
- AlgoTune
- Data Agent Benchmark
- SWE-Bench Pro V2
- Android Bench 2.0
- Vibe Code Bench v1
- Claw-SWE-Bench
- FeatureBench
- FeatureBench Lite
- MacroscopeBench
- DAYJOB: Data Science
- GameCraft-Bench
- Real-SWE 1.1
- SWE-bench Lite
- Next.js Agent Evals
- OTelBench
- SWE-rebench
- Tencent WorkBuddy Bench
- EvoCode-Bench
- EvoCode-Bench v2
- RoadmapBench
- SWE-bench-Live
- FDE-Gym
- Monthly-SWEBench
- SWE-Bench Pro Private
- ScarfBench
- Snorkel Agentic Coding
- GitTaskBench
- MiMo Visual Coding
- SWE-bench Science
- BIRD-CRITIC
- ContextBench
- MiMo Code Bench
- MMTU
- SWE-bench
- SWE-Clar
- SWT-bench
- ViBench 2.0
- Vision2Web S2-2026
- CodeClash
- CursorBench 3.1
- DSBench (DeepSeek internal)
- LibraryDesignBench
- QwenSWEBench
- ABC-Bench
- Kimi Code Bench 3.0 Long-horizon
- MiMo Coding Bench
- Multi-SWE-bench mini
- Vision2Web S3-2026
- AInsteinBench
- CooperBench
- Hy-SWE Max
- Kimi Code Bench v2
- Legacy-Bench
- LoopArena
- QwenQoderBench
- QwenReactBench
- StepCodeBench
- SWE-PolyBench
- SWE-Review
- TestPrism
- VIBE (MiniMax)
- Z.ai Code Bench
- CATArena Gomoku
- CATArena OIBench-5
- CATArena SWE-Perf-10
- CATArena Texas Holdem
- CC-Bench-V2 Backend
- CC-Bench-V2 Large Repo Exploration
- Multi-SWE-bench flash
- VIBE-Pro (MiniMax)
- Vision2Web S1-2026
- τ^τ-bench
- CC-Bench
- CC-Bench-V2 Frontend
- CursorBench 3.2
- Expert-SWE (OpenAI)
- Hy-Backend 2.0
- PRDBench
- ReviewBench
- SWE-Lancer Diamond (SWE Manager)
- SWE-Perf
- AgenticDataBench
- AInstein-SWE-Bench
- BIRD-Interact
- DevBench Implementation
- Internal agentic coding evaluation (Anthropic)
- Kernel Bench L3
- AA-3D
- CC-Bench-V2 Multi-Step Chained Tasks
- Code refactoring tasks (OpenAI)
- Code2Video-AA
- cognition-golden
- Cursor Bench (2025 internal version)
- Database Migration Tasks (OpenAI)
- DevBench Acceptance Testing
- DevBench Environment Setup
- DevBench Unit Testing
- DSBench
- Hy-Backend
- Hy-CompanyBench
- Hy-CompanyBench V2
- Hy-Vibe Bench
- HyperTask
- OpenRCA
- RepoBench
- Step-SWE-Bench
- SWE-bench-Live Verified
- SWE-Dev
- SWE-EVO
- VIBE-V2 (MiniMax)
- Coding Experience (Moonshot)
- Conversational SWE Task Benchmark (Windsurf)
- Data Analysis Benchmark (StepFun)
- DeepSeek R&D Coding Benchmark
- End-To-End SWE Task Benchmark (Windsurf)
- HaltEval-prelim (Meta)
- Kimi Webdev Bench
- SWE-bench Verified hard subset (Meta)
- AACR-Bench
- CATArena
- CATArena Pommerman
- CATArena Tetris
- CheckerBench
- Code Review Bench (Martian)
- ComputeEval
- CoreCodeBench
- DABstep
- DAComp
- Defects4J
- DevBench
- HiL-Bench
- Kimi Code Bench
- Qodo AI Code Review Benchmark
- Real-SWE
- SWE-Bench ProMax
- SWE-CI
- SWE-Explore
- SWE-Interact
- Trae Agent Bench
Research frontier
Researchers, on open problems or tasks that take days
Another 7 have no level. Levels don’t apply to arenas, safety evaluations or composite indices.
By kind
Function-level problems
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| HumanEval 164 hand-written Python function problems. Generated code is scored pass@1 against unit tests. | School | Checked automatically |  98.8% | Fading |
| MBPP Models write short Python functions from a one-line task description. A solution counts as correct if it passes the task's three assert tests. | School | Checked automatically | 92.7% | Fading |
| BigCodeBench-Hard Models solve 148 harder BigCodeBench Python tasks that call many libraries, chosen to match real programming work. Scored pass@1 on unit tests. | University | Checked automatically | 35.8% | Dropped |
| HumanEval+ Models write Python functions for the 164 HumanEval problems. EvalPlus checks them against about 80 times more tests than the original. | School | Checked automatically |  94.5% | Fading |
| BigCodeBench 1,140 practical Python tasks requiring calls to 139 libraries. Scored pass@1 on unit tests. | University | Checked automatically | 60.6% | Fading |
| Aider Polyglot Models edit code to solve 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust. Scored by the share passing unit tests. | University | Checked automatically | 88% | Fading |
| MBPP+ Models write short Python functions for MBPP problems, and EvalPlus checks each one against many more generated tests than the original. | School | Checked automatically |  86.2% | Fading |
| Aider-Edit Models edit Python source files to complete 133 Exercism exercises through Aider. Scored by the share of exercises whose unit tests all pass. | School | Checked automatically | 79.7% | Dropped |
| MultiPL-E Models write code for HumanEval and MBPP problems translated into 18 other programming languages. Scored by pass@1 on unit tests. | School | Checked automatically | 88.2% | Fading |
| AutoCodeBench Models solve automatically generated hard coding problems across many programming languages. A sandbox runs generated tests on each solution. | University | Checked automatically | 52.4% | Not in cards |
| CRUXEval Models predict the output of 800 short Python functions for a given input, or an input that yields a given output. Scored by pass@1. | School | Checked automatically | 95.9% | Fading |
| HumanEval-V Models read a diagram and a function signature, then write Python code that must pass test cases. Tasks span six types of visual reasoning. | University | Checked automatically | 40.6% | Fading |
| SEAL Coding Models answer 1,000 private coding prompts across common languages and tasks. Human annotators rate adherence, correctness, efficiency and readability, then compare responses side by side to produce Elo-style scores. | University | Graded by people | 1,237 | Not in cards |
| MHPP Models write Python functions for 210 human-curated problems that need careful reading of specifications and multi-step reasoning. Unit tests check each solution. | University | Checked automatically | 68.1% | Dropped |
| FullStackBench Models write code for 3,374 problems in 16 programming languages across domains like data analysis and web development. Unit tests score each answer. | University | Checked automatically | 68.3% | Fading |
| Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness. | School | Checked automatically | 94% | Dropped |
| BigCodeArena Users ask for code, both models' programs run in a sandbox, and users try the results before voting for the better answer. Votes give Elo ratings. | Everyday | Ranked by votes | 1,117 | Not in cards |
| HumanEval Multi Models complete HumanEval problems in Python and seven more languages, including C++, Java, PHP, and Bash. Unit tests score pass@1. | School | Checked automatically | 82.6% | Dropped |
| ts-bench v1 Coding agents solve 25 self-contained TypeScript exercises from Exercism. The score is the percentage of exercises that pass their tests. | University | Checked automatically | 96% | Not in cards |
| EvalPlus HumanEval+/MBPP+ with 80x more tests. Showed many 'passing' solutions were wrong. | School | Checked automatically | 87% | Dropped |
| NaturalCodeBench Models solve 402 Python and Java problems taken from real user queries to online coding services in 6 domains. Semi-automatically built tests check each solution. | University | Checked automatically | 48.3% | Dropped |
| CodeEditorBench Models edit code in four scenarios: debugging, translation, requirement switching and polishing. Unit tests check the edits, and polishing also measures speed and memory gains. | Competition | Checked automatically | 85.5% | System cards only |
| ts-bench Coding agents such as Claude Code, Codex and Gemini CLI solve TypeScript tasks, and tests decide pass or fail. Version 1 uses 25 Exercism practice exercises. Version 2 uses real tasks from a large TypeScript monorepo, taken from SWE-Lancer. | University | Checked automatically | 80% | Not in cards |
| TestPrism Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one. | Expert | Checked automatically | 28% | Not in cards |
| CanItEdit Models edit a given block of code to follow an instruction, such as adding a feature, fixing a described bug or changing the approach. Hidden tests check the edited code. | University | Checked automatically | 54.3% | Dropped |
| MBXP Models solve MBPP problems transpiled with their tests into more than 10 programming languages, such as Java, C++ and Go. Unit tests check each solution. | School | Checked automatically |  78.4% | Dropped |
| McEval Models complete, explain and generate code in 40 programming languages across about 16,000 test samples. Scored by test execution. | School | Checked automatically |  59.6% | System cards only |
| HumanEvalPack Models write functions for HumanEval problems translated into Python, JavaScript, Java, Go, C++ and Rust. Hidden tests give pass@1. | University | Checked automatically | 76.2% | System cards only |
| DS-Arena-Code Models answer common coding requests in an internal subjective test. GPT-4o judges responses against competitor models to give a win rate. | University | Graded by a model | 63.1% | Dropped |
| DS-FIM-Eval Models fill in missing code between a given prefix and suffix, as in editor plugin completion. | University | Checked automatically | 78.3% | Dropped |
| LBPP (Less Basic Python Problems) Models write Python solutions to 161 new problems built to avoid training-set contamination. Hidden tests score pass@1. | University | Checked automatically | 81% | System cards only |
| CodeUtilityBench (LG) Models handle 16 everyday coding tasks such as explaining, debugging, refactoring and testing code. A GPT-5 judge checks five rubrics per item. | University | Graded by a model |  71.9% | System cards only |
| Coding (MiniMax in-house) Models answer coding requests from real Hailuo AI users. The report does not describe the scoring. | University | Graded by a model | 90.2% | System cards only |
| HaltEval-prelim (Meta) Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds. | Expert | Checked automatically | 94% | System cards only |
| HumanEval Infilling Models fill in a masked middle span of HumanEval solutions given the code before and after it. Unit tests check the completed function. | School | Checked automatically | 84.8% | System cards only |
| HumanEval-COBOL (Cohere) Models write COBOL versions of HumanEval problems, or translate COBOL solutions to Python. Unit tests score pass@1. | University | Checked automatically | 25.3% | System cards only |
| ClassEval Models generate whole Python classes for 100 hand-built tasks. Tests check each method and the class as a whole. | University | Checked automatically | Not in cards | |
| CodeXGLUE A suite of 10 code tasks across 14 datasets, grouped as code to code, text to code, code to text and text to text. Tasks include clone detection, defect detection, code completion, code repair, code search, code generation and code summarization. Each task has its own metric, such as accuracy, BLEU or MRR. | School | Checked automatically | Not in cards | |
| Copilot Arena Developers inside their IDE see paired code completions from two models and pick one. Pairwise votes give an Elo-style ranking of code models. | Everyday | Ranked by votes | Not in cards | |
| QuixBugs Models repair 40 classic algorithm programs, in Python and Java, that each contain a one-line bug. The bugs fall into 14 defect classes. A fix counts when the program passes the test cases. | School | Checked automatically | Not in cards |
Competitive programming
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| LiveCodeBench v6 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. | Competition | Checked automatically | 93.5% | In use |
| LiveCodeBench Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. | Competition | Checked automatically | 93.5% | In use |
| ALE-Bench Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants. | Competition | Checked automatically | 3,164 | Not in cards |
| LiveCodeBench v5 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. | Competition | Checked automatically | 81.2% | Fading |
| Codeforces rating Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. | Competition | Checked automatically | 3,471 | In use |
| IOI (Vals) Agents solve International Olympiad in Informatics problems from 2024 and 2025 in C++. A submission tool modeled on the contest grades up to 50 tries per problem by subtask. | Competition | Checked automatically | 91.7% | Not in cards |
| IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run. | Competition | Checked automatically | 100% | Not in cards |
| OJBench Models write C++ or Python solutions to 232 competition problems from NOI and ICPC. Judge test cases score each solution by pass@1. | Competition | Checked automatically | 70% | Fading |
| AetherCode Models write solutions to problems from premier programming contests such as IOI and ICPC. Expert-validated test suites judge each solution. | Competition | Checked automatically | 35.5% | Fading |
| FrontierCS Models write programs for 156 open-ended computer science problems whose optimum is unknown. An automatic evaluator gives partial scores against an expert reference solution. | Research frontier | Checked automatically | 76.4 | System cards only |
| LiveCodeBench Pro Continuously updated Codeforces, ICPC and IOI problems annotated by olympiad medalists. Reported as Elo. | Competition | Checked automatically | 2,887 | Fading |
| Multi-LCB Models solve LiveCodeBench-style contest problems in twelve programming languages, with dated problems for contamination-aware evaluation. | Competition | Checked automatically | 67.8% | Not in cards |
| OIBench Models write programs for 250 original olympiad in informatics problems. Hidden tests check correctness, and time and space completion curves compare efficiency with reference solutions. | Competition | Checked automatically | 36.4% | Fading |
| CF-Div2-Stepfun Models solve 53 Codeforces Div. 2 problems from September 2024 to February 2025. An offline judge with rebuilt test cases scores C++, Python and Java submissions. | Competition | Checked automatically |  86.1% | System cards only |
| FrontierCS 1.1 Agents solve 179 algorithmic tasks under revised scoring that rewards solutions beyond the reference. The score is the official average task score. | Research frontier | Checked automatically | 56.5 | Not in cards |
| ICPC World Finals 2025 Models solve the 12 problems of the 2025 ICPC World Finals under contest time and attempt limits. Scored by problems solved. | Competition | Checked automatically | 10.0 | Fading |
| IOI 2025 Model solves the 2025 International Olympiad in Informatics tasks. Scored with official subtask points (max 600). | Competition | Checked automatically | 570.0 | Fading |
| APPS Models write Python programs for 10,000 coding problems that range from introductory to competition level. Generated code runs against test cases. | University | Checked automatically | 70.2% | Dropped |
| LeetCode Weekly Contest Models solve 126 LeetCode contest problems from July to November 2023. A solution counts only if it passes all of its 20+ test cases. | University | Checked automatically | 17.5% | Dropped |
| Leetcode (easy) Models solve 41 easy Leetcode problems that OpenAI selected. Scored as problems solved. | University | Checked automatically | 75.6% | System cards only |
| Leetcode (hard) Models solve 45 hard Leetcode problems that OpenAI selected. Scored as problems solved. | University | Checked automatically | 6.70% | System cards only |
| Leetcode (medium) Models solve 80 medium Leetcode problems that OpenAI selected. Scored as problems solved. | University | Checked automatically | 26.3% | System cards only |
| USACO Models solve 307 USA Computing Olympiad problems. Hidden unit tests check correctness and efficiency, scored as pass@1. | Competition | Checked automatically | 2.61% | Not in cards |
| CodeContests Models write programs for competitive programming problems, drawn from Codeforces and other sites, and the programs run against hidden tests. Google DeepMind built the dataset for AlphaCode, with training, validation (117 problems) and test (165 problems) splits and many generated tests per problem to cut false positives. Scores are the share of problems solved within a fixed number of submissions, such as 10@k. | Competition | Checked automatically | Not in cards |
Fixing real repositories
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| SWE-bench Verified 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. | Expert | Checked automatically | 83.5% | In use |
| SWE-Bench Pro 1,865 long-horizon issue-resolution tasks from 41 repos in four languages. Agent patch must pass hidden tests. Pass@1. | Expert | Checked automatically | 61.5% | In use |
| SWE-bench Multilingual 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. | Expert | Checked automatically | 96.6% | In use |
| DeepSWE 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. | Expert | Checked automatically | 74.1% | In use |
| Code Migration Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. | Expert | Checked automatically | 69.8% | Not in cards |
| APEX-SWE Software-engineering productivity tasks spanning integration work and observability, such as diagnosing failures from telemetry. | Expert | Checked automatically | 67.6% | In use |
| SWE Atlas Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. | Expert | Graded by a model | 63.2% | In use |
| SWE-bench Verified Extended Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores. | Expert | Checked automatically | 83.3% | Not in cards |
| FrontierCode 2.0 Agents resolve 132 maintainer-built tasks from deliberately ambiguous prompts. Rubric checks ask whether a maintainer would merge the change. | Expert | Checked automatically | 50.2% | In use |
| SWE Atlas Codebase QnA Extended Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores. | Expert | Graded by a model | 44.7% | Not in cards |
| AndroidBench Android development tasks from popular open-source apps: agent resolves issues or builds features, verified by test suites in emulator environments. | Expert | Checked automatically | 91.8% | In use |
| OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported. | Expert | Checked automatically | 81% | Not in cards |
| SWE-bench Multimodal SWE-bench tasks whose issues include screenshots or design mockups from visual JavaScript repos. Patch graded by hidden tests. | Expert | Checked automatically | 74.1% | In use |
| FrontierCode 1.1 Real-repo coding tasks graded for mergeability: correctness, regressions, lint, tests, scope and code quality. Failing any blocker scores zero. | Expert | Checked automatically | 53.5% | In use |
| Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. | Expert | Checked automatically | 86.9% | Not in cards |
| Multi-SWE-bench Agents resolve 1,632 GitHub issues in Java, TypeScript, JavaScript, Go, Rust, C, and C++ repos. Patches must pass the repos' tests. | Expert | Checked automatically | 52.7% | Fading |
| SWE Atlas v1.1 Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. | Expert | Graded by a model | 80.5% | Not in cards |
| CursorBench 4.0 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported. | Expert | Graded by a model | 60.4% | In use |
| SWE Atlas Test Writing Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests. | Expert | Graded by a model | 67% | In use |
| Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate. | Expert | Checked automatically | 39.3% | Not in cards |
| SWE-Bench Pro V2 Hard Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol. | Research frontier | Checked automatically | 98% | In use |
| CursorBench Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.0 covers edit, refactor and bugfix problems. Agentic graders score correctness, with cost reported. | Expert | Graded by a model | 60.4% | System cards only |
| SWE-Together Coding agents work through 109 repository tasks rebuilt from real multi-turn sessions between users and coding agents. A simulated user replays the original user's goals and gives feedback when the agent needs it. Scores count tasks solved and the number of corrections the user had to make. | Expert | Graded by a model | 69.3% | Not in cards |
| FrontierCode Real-repo coding tasks graded for mergeability: correctness, regressions, lint, tests, scope and code quality. Failing any blocker scores zero. | Expert | Checked automatically | 34.3% | In use |
| SWE-Lancer Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned. | Expert | Checked automatically | 81.4% | Fading |
| Brokk Power Ranking Models solve 181 real tickets from a 1.1 million line Rust codebase. Model judges compare solutions in pairs, and a Bradley-Terry fit gives each model a strength score. | Expert | Graded by a model | 39.7 | Not in cards |
| sudo L7 Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change. | Expert | Graded by people | 52.2% | In use |
| SWE Atlas Refactoring Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result. | Expert | Graded by a model | 59% | In use |
| SWE-Bench Pro V2 Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container. | Expert | Checked automatically | 99.4% | Not in cards |
| Claw-SWE-Bench General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches. | Expert | Checked automatically | 81.7% | Not in cards |
| FeatureBench Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically. | Expert | Checked automatically | 78.2% | Not in cards |
| FeatureBench Lite The public 30-task Lite leaderboard ranks passed-test percentages and reports fully resolved-task percentages separately. | Expert | Checked automatically | 78.2% | Not in cards |
| MacroscopeBench Models review commits from public repositories and must flag the known bugs a commit introduced without false reports. LLM judges match findings to the bugs. | Expert | Graded by a model | 83.9% | Not in cards |
| SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result. | Research frontier | Checked automatically | 16.6% | Not in cards |
| Real-SWE 1.1 Resolution rate on private enterprise coding tasks, averaged across five runs per task. | Expert | Checked automatically | 56.8% | Not in cards |
| SWE-bench Lite Models resolve 300 self-contained GitHub issues chosen from SWE-bench to cut evaluation cost. Hidden repository tests decide each patch. | Expert | Checked automatically | 60.3% | Not in cards |
| OTelBench Coding agents add OpenTelemetry tracing, metrics and logging to codebases in several languages. Tests check the instrumentation. | Expert | Checked automatically | 29% | Not in cards |
| SWE-rebench Agents fix fresh GitHub issues that an automated pipeline mines each month. Tests decide whether the issue is resolved. | Expert | Checked automatically | 64.5% | System cards only |
| Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search. | Expert | Checked automatically | 81.6% | Not in cards |
| SWE-bench-Live Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch. | Expert | Checked automatically | 71% | Dropped |
| Monthly-SWEBench Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1. | Expert | Checked automatically | 76% | Not in cards |
| SWE-Bench Pro Private Agents resolve held-out software issues from the private SWE-Bench Pro split. The leaderboard reports the percentage of resolved tasks. | Expert | Checked automatically | 51.5% | Not in cards |
| SWE-sweep Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved. | Research frontier | Checked automatically | 9.50% | In use |
| ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change. | Expert | Checked automatically | 75.2% | Not in cards |
| GitTaskBench Code agents solve 54 real tasks across 7 modalities and 7 domains by using a given GitHub repository, from understanding and setup to running it. | Expert | Checked automatically | 48.1% | Not in cards |
| ContextBench Coding agents resolve 1,136 issues from 66 repositories in eight programming languages, and the benchmark checks which code they look at along the way. Each task has human-annotated gold context, and the tracker scores context recall, precision and efficiency over the agent's trajectory. It adds these process measures to the usual issue-resolution result. | Expert | Checked automatically | 60.6% | Not in cards |
| MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details. | Expert | Checked automatically | 68.6% | In use |
| SWE-bench Models resolve 2,294 real GitHub issues from 12 Python repositories. A patch counts as resolved when the hidden repository tests pass. | Expert | Checked automatically | 52.6% | Not in cards |
| SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix. | Expert | Checked automatically | 89.1% | Fading |
| CursorBench 3.1 Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.1 added codebase understanding, bug finding, planning and code review problems. Agentic graders score correctness. | Expert | Graded by a model | 72.9% | Fading |
| QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. | Expert | Checked automatically | 86.3% | In use |
| ABC-Bench Coding agents take a real backend repository, edit the code, set up the environment and deploy the service in a container. The 224 tasks come from 127 open-source repositories in 8 languages and 19 frameworks. A task passes only when the running service passes external end-to-end API tests. | Expert | Checked automatically | 63.2% | Not in cards |
| MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE. | Expert | Checked automatically | 77.1% | Fading |
| Multi-SWE-bench mini Agents fix real GitHub issues in 8 programming languages, a 400-instance subset of Multi-SWE-bench made for cheaper runs. Scored by tests passing. | Expert | Checked automatically | 25.8% | Dropped |
| OctoCodingbench Agents complete 72 coding tasks while following rules from system prompts, AGENTS.md files, skills, and memory. An LLM judge checks 2,422 pass or fail items. | University | Graded by a model | 36.2% | Fading |
| Artificial Analysis Coding Agent Index v1.1 Version 1.1 of Artificial Analysis's coding agent index, which replaced SWE-Bench-Pro-Hard-AA with DeepSWE. | Checked automatically | 80.0 | In use | |
| CooperBench Two coding agents each get a different feature for the same repository and must coordinate so both changes work together. Over 600 tasks span 12 libraries in 4 languages. | Expert | Checked automatically | 27.9% | Not in cards |
| Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task. | Expert | Checked automatically | 64.2% | In use |
| Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages. | Expert | Checked automatically | 77.1% | In use |
| Legacy-Bench Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass. | Expert | Checked automatically | 60% | Not in cards |
| QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). | Expert | Checked automatically | 63.1% | In use |
| StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup. | Expert | Checked automatically | 63.9% | In use |
| SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution. | Expert | Checked automatically | 14.1% | Not in cards |
| SWE-Review Models review code changes in several languages and scenarios to find defects. Scores reflect defect recall and the rate of hallucinated defects. | Expert | Graded by a model | 16.2% | Fading |
| Z.ai Code Bench Zhipu in-house coding benchmark used for GLM-5.3. Reports pass rate and output tokens per task. Method unpublished. | Expert | Checked automatically | 39.5% | In use |
| CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. | Expert | Checked automatically | 25.8% | Fading |
| CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. | Expert | Checked automatically | 72.2% | Fading |
| Multi-SWE-bench flash Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing. | Expert | Checked automatically | 27% | Dropped |
| CursorBench 3.2 Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.2 added instruction following and advanced tool use problems. Agentic graders score correctness. | Expert | Graded by a model | 73.4% | In use |
| Hy-Backend 2.0 Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published. | Expert | Checked automatically | 35.2% | In use |
| KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages. | University | Checked automatically |  53.1% | In use |
| ReviewBench AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1. | Expert | Graded by a model | 73.3% | Not in cards |
| SWE-Lancer Diamond (SWE Manager) Models review competing implementation proposals for real freelance tasks and pick the best one. Scored against the original choice as pass@1. | Expert | Checked automatically | 51% | System cards only |
| Artificial Analysis Coding Agent Index v2 Coding agents, each a model plus harness, attempt software engineering, terminal, and repository question tasks. The index averages three equal-weight benchmark scores. | Checked automatically | 46.0 | In use | |
| Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass. | Expert | Checked automatically | 78% | System cards only |
| Artificial Analysis Coding Agent Index v1.4 Version 1.4 of Artificial Analysis's coding agent index, which moved to the full Terminal-Bench 2.1 and began scoring reward-hacked runs as zero. | Checked automatically | 67.0 | In use | |
| CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. | Expert | Checked automatically | 52.3% | System cards only |
| Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. | Expert | Checked automatically | 51.3% | Dropped |
| Cursor Bench (2025 internal version) Agents answer real agent requests from Cursor engineers in Cursor's harness. Graders compare against hand-curated solutions and check use of the codebase's existing abstractions. | Expert | Graded by a model | Fading | |
| Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | Expert | Checked automatically | 63.9% | In use |
| Hy-Backend Coding agents complete Tencent in-house backend development tasks. Task and scoring details are not published. | Expert | Checked automatically | 54.7% | Fading |
| RepoBench Models complete code using context retrieved from other files in the same Python or Java repository. Three tasks cover retrieval, completion and the full pipeline. | Expert | Checked automatically | 38% | Dropped |
| Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses. | Expert | Checked automatically | 67.1% | Fading |
| SWE-bench-Live Verified Task success on the official Verified split. The source reports each submission denominator separately. Other language and platform splits remain separate. | Expert | Checked automatically | 40% | Not in cards |
| SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail. | Expert | Checked automatically | 66.6% | Dropped |
| SWE-EVO Coding agents implement release-note changes in seven mature Python projects, editing about 21 files per task. Large test suites check each of the 48 tasks. | Expert | Checked automatically | 8.50% | Fading |
| Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability. | Expert | Graded by a model | 59.9% | System cards only |
| Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy. | Expert | Graded by a model | Dropped | |
| DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task. | Expert | Graded by people | 67% | System cards only |
| End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score. | Expert | Checked automatically | Dropped | |
| SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. | Expert | Checked automatically | 42.9% | System cards only |
| AACR-Bench Models review 200 real pull requests in many languages with full repository context. Their review comments are matched against expert-annotated issues. | Expert | Graded by a model | Not in cards | |
| Artificial Analysis Coding Agent Index v1.0 Artificial Analysis's first index of coding agents, each a model in a harness, averaged over three equally weighted coding benchmarks. | Checked automatically | Not in cards | ||
| Artificial Analysis Coding Agent Index v1.5 Version 1.5 of Artificial Analysis's coding agent index, which moved to the harder Terminal-Bench 4.0 and DeepSWE 1.1. | Checked automatically | Not in cards | ||
| CheckerBench Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback. | Expert | Checked automatically | Not in cards | |
| Code Review Bench (Martian) AI code review tools review pull requests from five large open-source projects. An LLM judge matches their comments to human-verified golden comments. | Expert | Graded by a model | Not in cards | |
| CoreCodeBench Models complete over 1,500 repository-level Python tasks covering development, bug fixing and test-driven work, with single-function and multi-function variants. The CorePipe pipeline builds tasks from repositories at set difficulty levels. | Expert | Checked automatically | Not in cards | |
| Defects4J A collection of 854 real, reproducible bugs from 17 open-source Java projects, each with the buggy and fixed versions and tests that expose the bug. Program repair studies ask a model to fix each bug, and a fix counts when the project's tests pass. It also serves fault localization and test generation research. | Expert | Checked automatically | Not in cards | |
| HiL-Bench Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary. | Expert | Checked automatically | Not in cards | |
| Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks. | Expert | Checked automatically | Not in cards | |
| Qodo AI Code Review Benchmark Models review real merged pull requests with planted functional bugs and best-practice violations. Findings are matched to the planted issues and scored by F1. | Expert | Checked automatically | Not in cards | |
| Real-SWE Models resolve tasks in private production codebases licensed from real companies. The board reports resolution rate with standard errors. | Expert | Checked automatically | Not in cards | |
| RepoChat Arena Users give a GitHub repository link and a question. A retriever model picks files, two answer models reply, and users vote for the better answer. | Everyday | Ranked by votes | Not in cards | |
| SWE-Bench ProMax Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions. | Expert | Checked automatically | Not in cards | |
| SWE-CI Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes. | Expert | Checked automatically | Not in cards | |
| SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail. | Expert | Checked automatically | Not in cards | |
| SWE-Interact Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session. | Expert | Checked automatically | Not in cards | |
| Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. | Expert | Checked automatically | System cards only |
Building whole projects
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| Vibe Code Bench End-to-end web application development from natural-language specs. Built apps are tested against functional requirements. | Expert | Graded by a model | 93.8% | In use |
| DesignArena Agentic Overall Coding agents build apps, games, slides and spreadsheets from user prompts. Users compare the results, and votes from every agent board form one rating. | Everyday | Ranked by votes | 1,442 | Not in cards |
| NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. | Expert | Checked automatically | 68.2% | In use |
| DesignArena React Native Models build cross-platform mobile apps in React Native from real user prompts. Users compare the running apps and vote for the better one. | Everyday | Ranked by votes | 1,321 | Not in cards |
| DesignArena Full Stack Coding agents build complete React apps with sign-in, databases and backend code from real user prompts. Users try the apps and vote for the better one. | Everyday | Ranked by votes | 1,393 | Not in cards |
| DesignArena Android (Kotlin) Models build native Android apps in Kotlin from real user prompts. The apps run in an Android emulator, and users compare them and vote. | Everyday | Ranked by votes | 1,397 | Not in cards |
| MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. | Research frontier | Checked automatically | 97.8% | Not in cards |
| ProgramBench almost solved Agents rebuild 200 programs from only their compiled binaries and docs. The score is the share of tasks whose rebuilt program passes at least 95% of the hidden behavioral tests. | Research frontier | Checked automatically | 70.5% | In use |
| ProgramBench 200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original. | Research frontier | Checked automatically | 18.5% | In use |
| ProgramBench tests passed Agents rebuild 200 programs from only their compiled binaries and docs. The score is the share of hidden behavioral tests the rebuilt program passes, averaged over tasks. | Research frontier | Checked automatically | 88.5% | In use |
| FrontierSWE 34 ultra-long-horizon engineering and research tasks. 5 trials of up to 20 hours each. Score is mean task score 0-100. | Research frontier | Checked automatically | 69.9% | In use |
| SWE-Marathon 20 realistic multi-hour software tasks (library reproductions, product clones, ML work). Verifiers combine hidden tests, parity and integrity checks. | Expert | Checked automatically | 29% | In use |
| DesignArena Agentic Game Dev Coding agents build multi-file games from a user prompt. Users play the final games and vote, and the logs keep tool calls, retries and failures. | Everyday | Ranked by votes | 1,265 | Not in cards |
| Vibe Code Bench 1-100 Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it. | Expert | Checked automatically | 34% | Not in cards |
| DesignArena Godot Game Dev Coding agents build complete Godot projects with GDScript, scenes and generated assets, exported to run in the browser. Users play the games and vote. | Everyday | Ranked by votes | 1,319 | Not in cards |
| FrontierSWE v1 Agents work up to 20 hours on 17 ultra-long engineering, performance and research tasks in their own harnesses. The board ranks models by average task rank. The score here is dominance, the win rate against a random baseline, which labs report. | Research frontier | Checked automatically | 88.2% | In use |
| SlopCodeBench Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity. | Expert | Checked automatically | 32.1% | Not in cards |
| GBAEval Agents get 24 hours to write a Game Boy Advance emulator in Rust for WebAssembly. Replay, hardware, and audio tests compare it with Mesen2. | Research frontier | Checked automatically | 79.6% | Not in cards |
| SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials. | Expert | Checked automatically | 53.8% | In use |
| Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. | Expert | Checked automatically | 40% | Not in cards |
| GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result. | University | Checked automatically | 70.6% | Not in cards |
| Vibe Code Bench v1 Agents build web applications from scratch from a natural-language spec in OpenHands. The built apps are tested against functional requirements. | Expert | Graded by a model | 41.3% | Not in cards |
| Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior. | Research frontier | Checked automatically | 47.6% | Not in cards |
| GameCraft-Bench Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together. | Expert | Checked automatically | 75.9% | Not in cards |
| OWE-Bench 26 long-horizon engineering and research projects across software, physical engineering, machine learning and scientific computing, from GPU kernels to aircraft design. Agents get 4 to 18 hours per attempt, and an offline verifier scores what they ship on held-out cases. | Research frontier | Checked automatically | 54.6% | Not in cards |
| EvoCode-Bench Coding agents keep one project working across 26 tasks of 5 to 15 rounds each, while requirements change, pile up and conflict. Cumulative hidden tests check every round in a persistent container. | Expert | Checked automatically | 54% | Not in cards |
| EvoCode-Bench v2 Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit. | Expert | Checked automatically | 59.1% | Not in cards |
| RoadmapBench Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests. | Expert | Checked automatically | 39.1% | In use |
| DesignArena Blender Agents build a 3D scene in Blender from a user prompt. Users compare the rendered scenes and vote for the better one. | Everyday | Ranked by votes | 1,413 | Not in cards |
| FDE-Gym Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1. | Expert | Checked automatically | 0.56 | Not in cards |
| Snorkel Agentic Coding Agents complete 100 multi-step coding tasks in sandboxed containers, spread evenly over four difficulty tiers. Unit tests and rubrics grade both outputs and the agent trajectory. | Expert | Checked automatically | 65.2% | Not in cards |
| SWE Refactor Bench Agents migrate 20 whole open-source repositories to a new language, framework, platform, or build system. Graded by migration audit, behavioral tests, and adversarial verifier models. | Research frontier | Checked automatically | 47% | In use |
| Web Clone Bench Agent gets a running reference app (Gmail, Jira, Figma, etc.) and 8 hours to build a clone. Expert-written workflows are replayed as end-to-end tests. | Research frontier | Checked automatically | 62% | Not in cards |
| SWE-Clar Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build. | Expert | Checked automatically | 67% | Not in cards |
| ViBench 2.0 Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans. | Expert | Graded by a model | 64.7% | Not in cards |
| CodeClash Models build and improve their own codebases over many rounds, then compete in arenas such as Halite, poker and BattleSnake. Results give Elo ratings. | Expert | Ranked by matches | 1,385 | Not in cards |
| DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring. | Expert | Checked automatically | 71.1% | In use |
| LibraryDesignBench An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity. | Expert | Checked automatically | 48.9 | Not in cards |
| Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score. | Expert | Checked automatically | 66.5% | In use |
| LoopArena Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit. | Expert | Checked automatically | 36.3% | Not in cards |
| VIBE (MiniMax) Agent builds complete web, simulation, Android, iOS or backend apps from scratch. Agent verifiers check execution, interaction and visuals. | Expert | Graded by a model | 90.7% | Fading |
| CATArena Gomoku Code agents compete in Gomoku. The public competition board reports Elo Rating for each agent configuration. | Expert | Ranked by matches | 1,799 | Not in cards |
| CATArena OIBench-5 Code agents compete in OIBench-5. The public competition board reports Time AUC for each agent configuration. | Expert | Ranked by matches | 0.40 | Not in cards |
| CATArena SWE-Perf-10 Code agents compete in SWE-Perf-10. The public competition board reports Time Ratio for each agent configuration. | Expert | Ranked by matches | 1.95× | Not in cards |
| CATArena Texas Holdem Code agents compete in Texas Holdem. The public competition board reports Elo Rating for each agent configuration. | Expert | Ranked by matches | 1,463 | Not in cards |
| VIBE-Pro (MiniMax) Agents build full-stack projects for Web, Android, iOS and Windows, including server APIs and databases. Claude Code based verifiers check interaction logic and visual output. | Expert | Graded by a model | 55.6% | Fading |
| τ^τ-bench Coding agent builds a customer-service agent from business records, a client and an API. The built agent is scored against held-out simulated users. | Expert | Checked automatically | 23.9% | Not in cards |
| CC-Bench Claude Code runs realistic frontend, backend, tooling and data coding tasks in fresh containers. Humans compare models head-to-head (win rate). | Expert | Ranked by votes | 41.9% | System cards only |
| Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. | Expert | Checked automatically | 83% | Fading |
| PRDBench Code agents build 50 real-world Python projects across 20 domains from product requirement documents. A trained judge agent checks each project against structured criteria. | Expert | Graded by a model | 69% | Not in cards |
| DevBench Implementation Models implement repository code. The score is the acceptance-test pass rate, with unit-test pass rate as an auxiliary metric. | Expert | Graded by a model | 7.10% | Not in cards |
| cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability. | Expert | Checked automatically | 51.8% | Not in cards |
| DevBench Acceptance Testing Models generate acceptance tests for repositories. The score is the published oracle-test percentage. | Expert | Graded by a model | 11.7% | Not in cards |
| DevBench Environment Setup Models prepare repository environments. The score is the percentage that pass the example usage test. | Expert | Graded by a model | 33.3% | Not in cards |
| DevBench Unit Testing Models generate unit tests for repositories. The score is the published oracle-test percentage. | Expert | Graded by a model | 28.7% | Not in cards |
| Hy-CompanyBench Coding agents complete Tencent in-house company engineering tasks. Task and scoring details are not published. | Expert | Checked automatically | 41.7% | In use |
| Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published. | Expert | Checked automatically | 62.4% | In use |
| Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published. | Expert | Checked automatically | 50.5% | Fading |
| HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials. | Expert | Graded by a model | 67.6% | System cards only |
| VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output. | Expert | Graded by a model | 50.1% | In use |
| CATArena Code agents write programs that play board and card games in repeated tournaments. Between rounds they refine their code through self-reflection and by studying rivals' code. Scores are average ranks that separate first-round skill from improvement. | Expert | Ranked by matches | Not in cards | |
| CATArena Pommerman Code agents compete in Pommerman. The public competition board reports Elo Rating for each agent configuration. | Expert | Ranked by matches | Not in cards | |
| CATArena Tetris Code agents compete in Tetris. The public competition board reports Score for each agent configuration. | Expert | Ranked by matches | Not in cards | |
| DesignArena Builder AI app builders make full-stack web and mobile apps from the same user prompt. Users compare the results and vote for the better one. | Everyday | Ranked by votes | Not in cards | |
| DesignArena Mobile Builder AI app builders make a mobile app from the same user prompt. Users compare the apps and vote for the better one. | Everyday | Ranked by votes | Not in cards | |
| DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design. | Expert | Graded by a model | Not in cards | |
| Surge AI blind coding evaluation Surge AI raters blindly score model coding outputs on a 1-5 scale. Mistral reports five models. | University | Graded by people | Not in cards |
Web and app front ends
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| DesignArena Website Models build a complete website as one HTML file from a real user prompt. Users compare the rendered pages and vote for the better one. | Everyday | Ranked by votes | 1,380 | In use |
| DesignArena Game Dev Models build a playable browser game as a single HTML file from a real user prompt. Users play the games and vote for the better one. | Everyday | Ranked by votes | 1,470 | In use |
| DesignArena Code Users give a prompt, and anonymous models each build a website, UI component, browser game, data visualization or 3D scene as one HTML file. Users vote, and votes become Bradley-Terry ratings. | Everyday | Ranked by votes | 1,396 | Not in cards |
| DesignArena UI Component Models build one interface component as a single HTML file from a real user prompt. Users compare the rendered components and vote for the better one. | Everyday | Ranked by votes | 1,401 | Not in cards |
| DesignArena 3D Design Models build an interactive 3D scene that runs in the browser as a single HTML file. Users explore the scenes and vote for the better one. | Everyday | Ranked by votes | 1,525 | In use |
| Code Arena Models act as coding agents that build web apps from user prompts, and users vote between two results. Votes yield Elo-style ratings. | Ranked by votes | 1,836 | In use | |
| WebDev Arena Two anonymous models build web apps from a user's prompt, and the user votes for the better one. Votes yield Elo-style ratings. | Ranked by votes | 1,820 | Fading | |
| DesignArena SVG Models write SVG code for a design request such as an icon, logo, badge or illustration. Users compare the rendered graphics and vote. | Everyday | Ranked by votes | 1,423 | Not in cards |
| Image-to-WebDev Arena Users upload a screenshot, wireframe or mockup, two anonymous models build the website in Code Arena, and users vote for the better result. Votes become ratings with bootstrapped confidence intervals. | Everyday | Ranked by votes | 1,752 | Not in cards |
| DesignArena Frontend Web App Coding agents build multi-file React frontends from real user prompts. Users compare the running apps and vote, and the run logs record agent behavior. | Everyday | Ranked by votes | 1,340 | In use |
| DesignArena Image to HTML Models recreate a reference image or mockup as a working single-file HTML website. Users compare the rendered pages and vote for the closer, better one. | Everyday | Ranked by votes | 1,304 | Not in cards |
| DesignArena Image to Frontend Coding agents turn a reference image into a multi-file frontend web app. Users compare the running apps and vote for the better one. | Everyday | Ranked by votes | 1,314 | Not in cards |
| ArtifactsBench Models generate interactive visual code artifacts for 1,825 tasks. An MLLM judge scores rendered screenshots and code against a per-task checklist. | University | Graded by a model | 66.8% | In use |
| ReactBench Coding agents do 39 realistic React tasks in two tracks. Write React tasks come from merged pull requests and are checked by hidden behavior tests. Fix React tasks ask the agent to remove React performance, accessibility and quality issues. A task passes only if the tests pass and a pinned React Doctor scan finds no new graded issue. | Expert | Checked automatically | 54.4% | Not in cards |
| Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Graded by a model | 82.3% | In use |
| OpenDesign Arena Models create design artifacts in the OpenDesign tool, and the board reports an evaluation score out of 100 with cost and speed per artifact. | Everyday | Graded by a model | 91.0 | Not in cards |
| DesignArena Video to Website Models watch a reference video and recreate its content as a working single-file HTML website. Users compare the rendered pages and vote. | Everyday | Ranked by votes | 1,250 | Not in cards |
| SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. | Everyday | Checked automatically | 0.92 | Fading |
| ArtifactsBench (Gemini 2.5 Pro 0506 judge) Models generate interactive visual code artifacts for 1,825 tasks. An MLLM judge scores rendered screenshots and code against a per-task checklist. | University | Graded by a model | 52.2% | Not in cards |
| ChartMimic Multimodal models write plotting code that reproduces 1,000 charts from scientific papers. Scores compare the rendered chart with the original. | University | Checked automatically | 60.3 | System cards only |
| Landing Page Arena Models create landing pages. Pairwise votes determine the published Arena rating. | Everyday | Ranked by votes | 1,692 | Not in cards |
| Next.js Agent Evals Coding agents with their models do real Next.js tasks. The board reports success rate with and without an AGENTS.md file, plus duration and cost. | Expert | Checked automatically | 100% | Not in cards |
| Design2Code Models turn screenshots of 484 real webpages into HTML and CSS. Automatic metrics compare the rendered page with the original on layout, text, position, color and CLIP similarity. | Everyday | Checked automatically | 87.2 | Fading |
| DesignArena Agentic Image to HTML Coding agents turn a user-uploaded image into a self-contained HTML website. Users compare how well each page reproduces the design and vote. | Everyday | Ranked by votes | 1,247 | Not in cards |
| MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code. | Expert | Graded by a model | 82.2% | In use |
| QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. | University | Graded by a model | 1,608 | Fading |
| Vision2Web S2-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Graded by a model | 43.5% | Not in cards |
| Flame-React-Eval Models write React components from UI design images. Rendered outputs are compared with gold components by image similarity and scored as pass@k. | University | Checked automatically | 93.8% | Fading |
| QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings. | University | Graded by a model | 1,532 | Fading |
| QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. | University | Graded by a model | 1,617 | Fading |
| QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. | University | Graded by a model | 69.9% | In use |
| Vision2Web S3-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Graded by a model | 74.6% | Not in cards |
| QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. | Expert | Graded by a model | 1,770 | In use |
| QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo. | University | Graded by a model | 1,758 | In use |
| QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. | University | Graded by a model | 1,884 | In use |
| Vision2Web S1-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Graded by a model | 52.7% | Not in cards |
| CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. | Expert | Graded by a model | 74.8% | Fading |
| AA-3D Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings. | Expert | Ranked by votes |  1,021 | In use |
| Code2Video-AA Artificial Analysis' implementation of HeyGen's Code2Video Bench: models write a HyperFrames HTML composition for each of 114 product and brand video briefs. A model judge compares two rendered videos at a time on five axes, and the comparisons become Elo ratings. | Expert | Graded by a model | 1,020 | In use |
| SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs. | University | Graded by a model | 63.7% | In use |
| U-Artifacts Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need. | Everyday | Graded by a model | 49.2% | Fading |
| Kimi Webdev Bench Models build games, 3D scenes and website clones from real prompts. Blind experts compare two outputs and report win, tie and loss rates. | Expert | Ranked by votes | 58.6% | System cards only |
Speed and optimization
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| GSO-Bench Agents speed up code in 102 tasks across 10 real codebases, given a performance test. Scored by the share matching the expert commit's speedup. | Expert | Checked automatically | 88.2% | Not in cards |
| SWE-fficiency 498 tasks in nine Python data/ML/HPC repos: agent must speed up a real workload without breaking tests. Scored as speedup relative to expert patch. | Expert | Checked automatically | 22.5% | In use |
| AlgoTune Agents rewrite code for 154 math, physics, and computer science functions to run faster and stay correct. Scoring is harmonic mean speedup over reference code. | Expert | Checked automatically | 2.05× | Not in cards |
| BigO(Bench) Models predict the time and space complexity of Python solutions and write solutions that meet a required complexity. Uses 3,105 Code Contests problems with profiled complexity labels. | Competition | Checked automatically | 41.4% | System cards only |
| SWE-Perf Agents speed up code in 140 tasks drawn from performance pull requests in real GitHub repos. Scored on correctness and runtime gain. | Expert | Checked automatically | 3.10% | Fading |
| Kernel Bench L3 Models write fast GPU kernels for the 50 full model architectures in KernelBench Level 3. Scored on correctness and speedup over PyTorch. | Expert | Checked automatically | Fading | |
| ComputeEval Models write CUDA code for handcrafted problems on kernels, runtime APIs and GPU libraries. Held-out test harnesses check correctness, and optional benchmarks time the code. | Expert | Checked automatically | Not in cards |
Data science and SQL
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| DesignArena Data Visualization Models turn a user prompt and its data into an interactive visualization as a single HTML file. Users compare the rendered charts and vote. | Everyday | Ranked by votes | 1,388 | Not in cards |
| DAXBench v2 Models write DAX, the formula language of Power BI, for 94 tasks on understanding, writing and reasoning over the Contoso sample model. Each answer runs inside real DAX queries in several report contexts, such as detail rows, subtotals, the grand total and slicers. Harder tasks earn more points, with bonuses for best practices and efficient code. | University | Checked automatically | 95.9% | Not in cards |
| LiveSQLBench Contamination-resistant text-to-SQL on PostgreSQL with hierarchical knowledge bases, including CRUD tasks. Scored by soft execution match and test cases. | University | Checked automatically | 40.2% | In use |
| Bird-SQL Models convert natural language questions into SQL over 95 large real databases across 37 domains. Scored by execution accuracy. | University | Checked automatically | 63.6% | Fading |
| StructEval Models generate or convert structured outputs in 18 formats, such as JSON, TOML and HTML. StructEval-T covers text-only formats and StructEval-V renderable ones. | Everyday | Checked automatically | 81.2% | In use |
| Spider 2.0 Models and agents write SQL for real enterprise databases with large schemas, across BigQuery, Snowflake, SQLite and dbt projects. Answers are checked by executing them. | Expert | Checked automatically | 94.2% | Dropped |
| Data Agent Benchmark Agents answer business questions over data split across several database systems, with duplicated and inconsistent records and key facts buried in text. The 104 queries cover 17 datasets and four database systems, based on a study of companies building data agents in six industries. Scores are pass@1 against known answers. | Expert | Checked automatically | 99.2% | Not in cards |
| Chart Arena Models create charts. Pairwise votes determine the published Arena rating. | Everyday | Ranked by votes | 1,628 | Not in cards |
| DAYJOB: Data Science Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Graded by a model | 37.3% | Not in cards |
| Floorplan Arena Models furnish hand-drawn room sketches. People compare the layouts and vote for the better floorplan. | Everyday | Ranked by votes | 1,792 | Not in cards |
| BIRD-CRITIC Models fix real user SQL issues in MySQL, PostgreSQL, SQL Server and Oracle applications. Test cases check each fix. | Expert | Checked automatically | 35.5% | Not in cards |
| MMTU Models answer about 30,000 questions over real tables across 25 tasks, such as data cleaning, table joins, column and table transformation, table question answering and turning language into code. The tasks come from decades of computer science research on tabular data. Answers are checked automatically against references. | Expert | Checked automatically | 69.6% | Not in cards |
| Spider Models turn natural language questions into SQL over 200 multi-table databases from 138 domains. Test databases are unseen in training, and answers are scored by exact match or execution. | University | Checked automatically | 83.7% | Dropped |
| DS-1000 Models write Python for 1,000 data science problems from StackOverflow across seven libraries such as NumPy and Pandas. Test cases and API constraints check each answer. | University | Checked automatically | 53.9% | Not in cards |
| AgenticDataBench Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios. | Expert | Checked automatically | 48.8% | Not in cards |
| BIRD-Interact Models solve ambiguous database tasks by talking with a simulated user, in a conversational mode and an agentic mode. 600 BI and CRUD tasks are checked by executable tests. | Expert | Checked automatically | 25.5% | Not in cards |
| DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap. | Expert | Checked automatically | 89.9% | Dropped |
| OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason. | Expert | Checked automatically | 34.9% | System cards only |
| Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. | Expert | Graded by a model | 39.6% | System cards only |
| Multi-dialect SQL evaluation (Cohere) Models write SQL for SQLite, PostgreSQL, MySQL, PL/SQL and T-SQL databases. Scored by execution accuracy against gold databases. | University | Checked automatically | 55.3% | System cards only |
| DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. | Expert | Checked automatically | Not in cards | |
| DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. | Expert | Graded by a model | Not in cards | |
| DataBench Models answer 1,822 questions about 80 real-world tables, such as Forbes billionaires and Airbnb listings, with 3.7 million rows in total. Answers are booleans, numbers, categories or lists, matched against a key. A lite setting gives each table as a small sample. | University | Checked automatically | Not in cards | |
| WikiSQL Models turn natural language questions into SQL queries over single Wikipedia tables. The dataset has 80,654 hand-annotated questions over 24,241 tables. Scored by execution accuracy and logical form accuracy. | School | Checked automatically | Not in cards |
Scientific code
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. | Expert | Checked automatically | 59.8% | In use |
| SWE-bench Science Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix. | Expert | Checked automatically | 50.4% | Not in cards |
| AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. | Expert | Checked automatically | 42.8% | Not in cards |
| AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. | Expert | Checked automatically | 47.7% | Fading |
| ArXivCS Models answer research-level computer science questions derived from recent arXiv papers, each with a reference final answer. The first set has 46 questions from August 2026. | Research frontier | Checked automatically | Not in cards |