Benchmarks
We track 6,674 benchmarks, with the last one added on November 20th. Arenas, where models are scored with Elo rankings or by human voting, can be found here. Have a benchmark that you want us to track? Let us know here.
Scores refreshed November 21st, 2026
- All areas
- Knowledge and language 1403
- Knowledge 341
- Exam questions 48
- Expert-written questions 19
- Commonsense 21
- Reading comprehension 75
- Trivia and open questions 17
- Questions that need search 71
- Language reasoning 90
- Factuality 172
- Short factual questions 30
- Faithful to sources 66
- Long-form factuality 28
- Hallucination rates 48
- Multilingual 589
- Knowledge in many languages 126
- Translation 80
- Reasoning in many languages 91
- Reading in many languages 255
- Chat and instructions in many languages 37
- Long context 120
- Finding facts in long inputs 35
- Reasoning over long documents 58
- Learning from the context 26
- Instructions and chat 127
- Following precise instructions 65
- Multi-turn conversation 30
- Overall chat quality 30
- Writing and taste 54
- Creative writing 33
- Taste and style 21
- Math and science 412
- Math 157
- Word problems 34
- Competition problems with answers 54
- Written proofs 22
- Formal proofs 25
- Research mathematics 22
- Science and reasoning 206
- Science questions 44
- Science olympiads 9
- Research science 78
- Lab and data work 50
- Abstract puzzles and logic 25
- Forecasting 49
- Real-world events 32
- Social simulation 16
- Coding 679
- Coding 508
- Function-level problems 70
- Competitive programming 29
- Fixing real repositories 211
- Building whole projects 71
- Web and app front ends 42
- Speed and optimization 15
- Data science and SQL 44
- Scientific code 26
- AI research 171
- Machine learning engineering 59
- AI research tasks 63
- GPU kernels 23
- Task length 9
- Labs' internal R&D tests 17
- Agents and tools 639
- Terminal agents 41
- Terminal tasks 32
- Long terminal sessions 9
- Computer and browser use 128
- Desktop apps 30
- Phones 38
- Web browsing 42
- Finding things on screen 18
- Tool use 289
- Function calling 53
- Customer service 49
- Apps and MCP servers 60
- Deep research 62
- General assistants 65
- Games and long tasks 181
- Puzzles and chess 72
- Video games 61
- Multi-agent games 48
- Professional work 1012
- Economic work 216
- Tasks from real jobs 80
- Running a business 50
- Documents and spreadsheets 53
- Simulated companies 18
- Composite work indices 15
- Law, finance and medicine 681
- Law 124
- Finance 181
- Medicine 295
- Other professions 81
- Engineering and CAD 115
- CAD models 47
- Electronics 68
- Vision, audio and robotics 1528
- Vision 620
- General visual questions 215
- Documents, charts and text in images 128
- Visual math and diagrams 19
- Spatial reasoning and counting 132
- Detection and captioning 59
- Image generation and editing 66
- Audio and video 676
- Video understanding 168
- Long videos 27
- Video captioning 27
- Speech recognition and translation 192
- Audio understanding and voice 209
- Video generation 53
- Robotics 176
- Simulated robots 99
- Real robots 30
- Embodied reasoning 47
- World models 56
- Video world models 17
- Interactive world models 20
- Physical prediction 19
- Safety and security 923
- Safety and alignment 633
- Refusing harmful requests 167
- Jailbreaks and prompt injection 124
- Deception and alignment 212
- Bias, fairness and politics 59
- Wellbeing and sycophancy 44
- Dangerous capabilities 25
- Cybersecurity 145
- Capture-the-flag challenges 44
- Finding and exploiting real bugs 71
- Secure code and review 30
- Biology and chemistry 145
- Lab protocols 13
- Genomics and life sciences 91
- Hazardous knowledge 41
- Other 78
- Other 78
- Composite indices 53
- Other 24
Nothing matches.
- Any level
- Everyday 2237
- School 602
- University 1520
- Competition 139
- Expert 1878
- Research frontier 188
- Any status
- In use 480
- Fading 364
- Dropped 221
- Only in system cards 875
- Not in model cards 4734
- Last model: any time
- New model within 3 months 1439
- New model within 6 months 2041
- New model within a year 2903
- Quiet for a year or more 1401
- No scores yet 1993
- With or without code
- Public code 3643
- Cited: all time
- Cited: last 12 months
- Cited: last 6 months
What it measures
- Measures
- Capability 5888
- Propensity 656
- Preference 80
- Efficiency 50
- Risk
- None 5953
- CBRN 60
- Cyber offence 233
- Loss of control 154
- Harmful manipulation 38
- Violent crimes 73
- Non-violent crimes 86
- Sex-related crimes 14
- Child sexual exploitation 25
- Defamation 15
- Specialized advice 77
- Privacy 124
- Intellectual property 17
- Indiscriminate weapons 53
- Hate 92
- Suicide and self-harm 55
- Sexual content 46
- Elections 40
Nothing matches.
- Behavior
- None 6034
- Misaligned with intent 183
- Misaligned with values 45
- Harmful use 254
- Hallucination 118
- Bias 68
- Unreliability 10
- Lawlessness 37
- Power-seeking 25
- Collusion 4
- Mis-coordination 1
What the tasks are about
- Subject
- Natural sciences 2082
- Mathematics 432
- Computer and information sciences 1389
- Physical sciences 232
- Chemical sciences 126
- Earth and environmental sciences 74
- Biological sciences 335
- Other natural sciences 15
- Engineering and technology 359
- Civil engineering 30
- Electrical, electronic and information engineering 225
- Mechanical engineering 73
- Chemical engineering 4
- Materials engineering 28
- Medical engineering 4
- Environmental engineering 5
- Environmental biotechnology 2
- Nanotechnology 3
- Other engineering and technologies 20
- Medical and health sciences 439
- Basic medicine 54
- Clinical medicine 350
- Health sciences 50
- Medical biotechnology 15
- Agricultural and veterinary sciences 30
- Agriculture, forestry and fisheries 27
- Animal and dairy science 1
- Veterinary science 2
- Agricultural biotechnology 1
- Social sciences 843
- Psychology 45
- Economics and business 423
- Education 69
- Sociology 10
- Law 225
- Political science 65
- Social and economic geography 30
- Media and communications 35
- Other social sciences 34
- Humanities and the arts 678
- History and archaeology 65
- Languages and literature 459
- Philosophy, ethics and religion 36
- Arts 143
- Other humanities 14
- No specialist field 2919
Nothing matches.
- Language
- English 5327
- Chinese 354
- Many languages 317
- Not stated 294
- Arabic 66
- Japanese 60
- Korean 49
- Spanish 43
- French 42
- German 39
- Portuguese 38
- Icelandic 25
- Russian 24
- Swedish 21
- Hindi 17
- Vietnamese 14
- Catalan 13
- Norwegian 13
- Persian 13
- Danish 12
- Italian 12
- Latin 11
- Dutch 10
- Estonian 10
- Hebrew 10
- Basque 9
- Bangla 8
- Faroese 8
- Greek 8
- Polish 8
- Finnish 7
- Sanskrit 7
- Czech 6
- Swahili 6
- Amharic 5
- Norwegian Bokmål 5
- Turkish 5
- Yoruba 5
- Hausa 4
- Hungarian 4
- Igbo 4
- Norwegian Nynorsk 4
- Tamil 4
- Thai 4
- Urdu 4
- Uzbek 4
- Ancient Greek 3
- Indonesian 3
- Kazakh 3
- Telugu 3
- Bhojpuri 2
- Bulgarian 2
- bzd 2
- Croatian 2
- Galician 2
- Guarani 2
- Gujarati 2
- Kannada 2
- Khmer 2
- Nigerian Pidgin 2
- Nyanja 2
- Serbian 2
- Sinhala 2
- Slovak 2
- Zulu 2
- Akkadian 1
- Armenian 1
- Belarusian 1
- Bemba 1
- Bosnian 1
- Cantonese 1
- Chuvash 1
- ecs 1
- Ewe 1
- inl 1
- ins 1
- kgv 1
- Kurdish 1
- Kyrgyz 1
- Latvian 1
- Literary Chinese 1
- Lithuanian 1
- Luxembourgish 1
- Malayalam 1
- Maltese 1
- Māori 1
- nah 1
- Nepali 1
- nhi 1
- Odia 1
- Slovenian 1
- Sumerian 1
- Tulu 1
- Ukrainian 1
- Uyghur 1
- wbs 1
- Wolof 1
- xml 1
- yua 1
Nothing matches.
What goes in and out
- Input
- Text 6076
- Code 937
- Images 1038
- Video 371
- Audio 480
- 3D 52
- Tables 225
- Documents 315
- Screens 207
- Molecules and sequences 79
- Time series 166
- Maps and geodata 50
- Output
- Text 4825
- Code 1101
- Images 168
- Video 100
- Audio 143
- 3D 83
- Tables 101
- Documents 110
- Molecules and sequences 28
- Time series 29
- Maps and geodata 16
- Actions 1991
Conditions
- Interaction
- Single turn 4261
- Multi-turn 222
- Agentic 2008
- Multi-agent 183
- Environment
- None 4475
- Code sandbox 216
- Terminal 750
- Browser 276
- Desktop apps 90
- Phone apps 42
- APIs and tools 560
- A named app 126
- A named game 119
- A simulator 362
- A physical robot or device 56
- The live world 34
Origin and upkeep
- Items from
- Written by experts 433
- Written by crowdworkers 145
- Exams and contests 365
- Real work and use 2170
- Another benchmark 1026
- Translated 126
- Generated 903
- Live play or votes 84
- Not stated 2343
- Refresh
- One fixed set 5585
- New versions replace it 949
- New items on a schedule 47
- Live 79
- Not stated 14
- Run by
- Anyone 4574
- A host 1329
- A model developer 771
- Based on
- Its own tasks 5240
- A version of a public benchmark 1426
- A run of another’s method 8
Facets describe each benchmark by a fixed list of values. How we tag them, or open a saved view.
| GPQA Diamond 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. | NYU (Rein et al.) | Science questions | Expert | In use | | 258 | 168 | Nov 2023 | Nov 2026 |
| AIME 2026 Models solve the 30 integer-answer problems from the 2026 AIME I and II math competitions. Scored by exact answer match. | Mathematical Association of America (exam), MathArena (evaluation) | Competition problems with answers | Competition | In use | GPT-5.5 100% | 35 | 23 | Feb 2026 | Oct 2026 |
| AIME 2025 Replaced in Feb 2026 | Mathematical Association of America (MAA) | In use | Grok 4 Heavy 100% | 129 | 81 | Feb 2025 | Aug 2026 | ||
| AIME 2024 Replaced in Feb 2025 | Mathematical Association of America | Fading | Claude Opus 4.5 99.6% | 70 | 57 | Feb 2024 | Apr 2026 | ||
| AIME 2023 Replaced in Feb 2024 | Mathematical Association of America | Not in cards | Feb 2023 | ||||||
| Terminal-Bench 4.1 The Terminal-Bench 4.0 tasks with hardened verifiers: agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. | Stanford University, Harbor, and Laude Institute | Terminal tasks | Expert | In use | Copilot Fusion 71.2% | 2 | 1 | Nov 2026 | Nov 2026 |
| Terminal-Bench 4.0 Replaced in Nov 2026 | In use | Claude Opus 5.5 64.8% | 29 | 15 | Aug 2026 | Nov 2026 | |||
| Terminal-Bench 3.0 Replaced in Aug 2026 | Laude Institute and Harbor | In use | Claude Opus 5 42.7% | 6 | 4 | Jul 2026 | Sep 2026 | ||
| Terminal-Bench 2.1 Replaced in Jul 2026 | Stanford + Laude Institute | In use | GPT-6 Astra 87.4% | 59 | 23 | May 2026 | Nov 2026 | ||
| Terminal-Bench 2.0 Replaced in May 2026 | Stanford + Laude Institute | In use | GPT-5.5 82.3% | 62 | 28 | Nov 2025 | Jul 2026 | ||
| Terminal-Bench 1.0 Replaced in Nov 2025 | Stanford University and Laude Institute | Fading | GPT-5 49.6% | 20 | 8 | May 2025 | Dec 2025 | ||
| LiveCodeBench v6 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. | UC Berkeley / MIT / Cornell | Competitive programming | Competition | In use | | 76 | 53 | Apr 2025 | Oct 2026 |
| LiveCodeBench v5 Replaced in Apr 2025 | Fading | | 36 | 32 | Jan 2025 | Apr 2026 | |||
| LiveCodeBench v1 to v4 Replaced in Jan 2025 | In use | | 53 | 37 | Mar 2024 | Oct 2026 | |||
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Center for AI Safety and Scale AI | Expert-written questions | Research frontier | In use | Claude Fable 5.2 77.9% | 5 | 2 | Sep 2026 | Nov 2026 |
| HLE-Rolling Replaced in Sep 2026 | Not in cards | Oct 2025 | |||||||
| Humanity's Last Exam Original Replaced in Oct 2025 | Center for AI Safety + Scale AI | Expert | In use | GPT-6 Astra 54.8% | 145 | 73 | Jan 2025 | Nov 2026 | |
| SWE-bench Verified 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. | Princeton NLP + OpenAI | Fixing real repositories | Expert | In use | Claude Opus 4.7 83.5% | 144 | 82 | Aug 2024 | Nov 2026 |
| MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. | TIGER-Lab (Waterloo) | Exam questions | University | In use | | 139 | 103 | Jun 2024 | Oct 2026 |
| τ-bench 2.0 2.0.0 Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture. | Sierra | Customer service | University | Not in cards | GPT-Live-1 79.9% | Oct 2026 | Sep 2026 | ||
| τc-Bench Contextual Replaced in Oct 2026 | Technion and IBM Research | Everyday | Not in cards | | May 2026 | Dec 2025 | |||
| tau3-bench Replaced in May 2026 | Everyday | In use | Mercury 2.5 96% | 14 | 1 | Mar 2026 | Oct 2026 | ||
| tau2-bench Replaced in Mar 2026 | Sierra Research | Everyday | In use | GPT-5.4 98.9% | 83 | 40 | Jun 2025 | Oct 2026 | |
| tau-bench v1 Replaced in Jun 2025 | Everyday | Dropped | Claude Opus 4 70.5% | 21 | 10 | Jun 2024 | Sep 2025 | ||
| IFEval Models answer about 500 prompts containing verifiable instructions such as word limits or required keywords. Code checks each instruction for strict and loose accuracy. | Following precise instructions | Everyday | In use | | 103 | 88 | Nov 2023 | Oct 2026 | |
| MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. | UC Berkeley (Hendrycks et al.) | Exam questions | University | In use | | 100 | 91 | Sep 2020 | Oct 2026 |
| MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. | MMMU team (OSU, Waterloo and others) | General visual questions | University | In use | | 96 | 51 | Nov 2023 | Oct 2026 |
| MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. | MMMU team | General visual questions | University | In use | GPT-5.6 Sol 84.6% | 84 | 46 | Sep 2024 | Nov 2026 |
| OSWorld 2.1 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. | XLANG Lab, HKU | Desktop apps | University | In use | Claude Opus 5 44.3% | 8 | 5 | Sep 2026 | Nov 2026 |
| OSWorld 2.0 (Aug 2026) 2.0 Aug Replaced in Sep 2026 | System cards only | Claude Fable 5.1 77.9% | 1 | Aug 2026 | Oct 2026 | ||||
| OSWorld 2.0 Replaced in Aug 2026 | In use | Claude Opus 4.8 20.6% | 18 | 6 | Jun 2026 | Oct 2026 | |||
| OSWorld-Verified Replaced in Jun 2026 | XLANG Lab (HKU) + collaborators | Everyday | In use | Claude Fable 5 86% | 40 | 19 | Jul 2025 | Oct 2026 | |
| OSWorld Original Replaced in Jul 2025 | Everyday | Fading | | 13 | 10 | Apr 2024 | Jun 2026 | ||
| BrowseComp Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. | OpenAI | Web browsing | Expert | In use | Claude Fable 5 93.3% | 79 | 35 | Apr 2025 | Nov 2026 |
| SWE-Bench Pro V2 2.0 Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container. | Scale AI (co-developed with Reflection) | Fixing real repositories | Expert | In use | Claude Opus 5 99.4% | 1 | Sep 2026 | Nov 2026 | |
| SWE-Bench Pro 1.0 Replaced in Sep 2026 | Scale AI | In use | | 77 | 37 | Sep 2025 | Nov 2026 | ||
| IFBench 58 new out-of-domain verifiable output constraints. Scored by programmatic constraint checks (strict or loose). | AllenAI | Following precise instructions | Everyday | In use | Flint 89.2% | 69 | 43 | Jul 2025 | Oct 2026 |
| BFCL V4 Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. | UC Berkeley (Gorilla) | Function calling | Everyday | In use | | 32 | 11 | Jul 2025 | Oct 2026 |
| Berkeley Function Calling Leaderboard (BFCL) V1 to V3 Replaced in Jul 2025 | In use | | 36 | 31 | Feb 2024 | Oct 2026 | |||
| SWE-bench Multilingual 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. | SWE-bench team (Princeton/Stanford) | Fixing real repositories | Expert | In use | Claude Fable 5.2 96.6% | 67 | 39 | Apr 2025 | Nov 2026 |
| CharXiv Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. | Princeton NLP | Documents, charts and text in images | University | In use | Claude Fable 5 93.5% | 67 | 35 | Jun 2024 | Nov 2026 |
| HMMT 2026 Models solve the 33 final-answer problems from the February 2026 Harvard-MIT Mathematics Tournament. Scored by exact answer match. | HMMT (exam), MathArena (evaluation) | Competition problems with answers | Competition | In use | GPT-5.5 98.5% | 18 | 7 | Feb 2026 | Nov 2026 |
| HMMT 2025 Replaced in Feb 2026 | Harvard-MIT Mathematics Tournament (HMMT) | Fading | | 49 | 25 | Feb 2025 | Aug 2026 | ||
| HMMT February 2024 Replaced in Feb 2025 | Harvard-MIT Mathematics Tournament (HMMT) | Not in cards | Feb 2024 | ||||||
| HumanEval 164 hand-written Python function problems. Generated code is scored pass@1 against unit tests. | OpenAI | Function-level problems | School | Fading | | 65 | 70 | Jul 2021 | Oct 2026 |
| MathVista Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. | UCLA + Microsoft | Visual math and diagrams | School | In use | | 65 | 42 | Oct 2023 | Oct 2026 |
| MATH 12,500 competition math problems with step-by-step solutions. Final answer scored by exact match. | UC Berkeley (Hendrycks et al.) | Competition problems with answers | Competition | Fading | | 64 | 72 | Mar 2021 | Oct 2026 |
| GSM8K 8.5K grade-school math word problems. Final numeric answer scored by exact match. | OpenAI | Word problems | School | Fading | | 57 | 72 | Oct 2021 | Oct 2026 |
| MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. | OpenAI | Knowledge in many languages | University | In use | Claude Mythos Preview 92.7% | 56 | 42 | Sep 2024 | Aug 2026 |
| SimpleQA Verified 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. | Google DeepMind | Short factual questions | University | In use | | 13 | 20 | Sep 2025 | Nov 2026 |
| SimpleQA Original Replaced in Sep 2025 | OpenAI | Fading | | 42 | 40 | Oct 2024 | Apr 2026 | ||
| OCRBench v2 Visual-text localization and reasoning tasks across 31 scenarios (documents, scenes, charts). Averaged task accuracy. | HUST (Yuliang Liu et al.) | Documents, charts and text in images | Everyday | In use | | 20 | 16 | Dec 2024 | Oct 2026 |
| OCRBench v1 Replaced in Dec 2024 | Yuliang Liu, Xiang Bai et al. | In use | GLM-OCR 94% | 35 | 20 | May 2023 | Oct 2026 | ||
| DeepSWE 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. | Datacurve | Fixing real repositories | Expert | In use | GPT-6 Astra 74.1% | 54 | 21 | May 2026 | Nov 2026 |
| AutomationBench Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. | Zapier | Apps and MCP servers | University | In use | | 54 | 19 | Apr 2026 | Nov 2026 |
| VideoMME v2 (with subtitles) Models answer multiple choice questions about videos with subtitles or audio added. Related questions are scored as groups to penalize guessing. | Video-MME team (Chaoyou Fu et al.) | Video understanding | Everyday | In use | | 6 | 3 | Apr 2026 | Oct 2026 |
| Video-MME v1 Replaced in Apr 2026 | Video-MME team (USTC, XMU, et al.) | In use | | 47 | 27 | May 2024 | Nov 2026 | ||
| GDPval-AA v2.1 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. | Artificial Analysis | Tasks from real jobs | Expert | In use | Claude Opus 5.5 1,846 | 11 | 6 | Sep 2026 | Nov 2026 |
| GDPval-AA v2 Replaced in Sep 2026 | In use | Claude Opus 5 1,861 | 26 | 18 | Jun 2026 | Oct 2026 | |||
| GDPval-AA v1 Replaced in Jun 2026 | Fading | Claude Fable 5 1,932 | 15 | 14 | Dec 2025 | Jun 2026 | |||
| MATH-500 Models solve 500 competition math problems drawn from the MATH test set. Final answers are checked against the reference answers. | OpenAI | Competition problems with answers | Competition | Fading | | 50 | 46 | May 2023 | Oct 2026 |
| BIG-Bench Hard Models solve 23 BIG-Bench tasks where earlier models trailed average human raters, such as logic puzzles and date math. Scoring is exact match. | Suzgun et al. (Stanford University and Google Research) | Language reasoning | School | Fading | | 49 | 59 | Oct 2022 | Oct 2026 |
| MATH-Vision (MathVision) 3,040 competition math problems with visual context. Answer accuracy. | CUHK MMLab | Visual math and diagrams | Competition | In use | | 48 | 32 | Feb 2024 | Nov 2026 |
| Arena-Hard v2.0 Models answer 500 hard real user prompts from Chatbot Arena and 250 creative writing prompts. GPT-4.1 or Gemini 2.5 judges each answer against a baseline answer from o3-mini, and the score is the win rate. | Arena (formerly LMArena) | Overall chat quality | University | Fading | | 20 | 14 | Apr 2025 | Aug 2026 |
| Arena-Hard-Auto 0.1 Replaced in Apr 2025 | Fading | | 27 | 25 | Apr 2024 | Apr 2026 | |||
| ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. | ARC Prize Foundation | Puzzles and chess | Everyday | In use | GPT-6.2 Astra 100% | 5 | 2 | Mar 2026 | Nov 2026 |
| ARC-AGI-2 Replaced in Mar 2026 | Abstract puzzles and logic | In use | GPT-6.2 Astra 95.8% | 24 | 13 | Mar 2025 | Nov 2026 | ||
| ARC-AGI v1 Replaced in Mar 2025 | ARC Prize Foundation (Chollet) | Abstract puzzles and logic | In use | Claude Fable 5 98.5% | 16 | 16 | Nov 2019 | Nov 2026 | |
| AI2D Models answer multiple choice questions about grade school science diagrams. The set has about 5,000 diagrams and 15,000 questions, scored by accuracy. | Allen Institute for AI (AI2) | Visual math and diagrams | School | Fading | | 44 | 30 | Mar 2016 | Aug 2026 |
| MCP Atlas Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. | Scale AI | Apps and MCP servers | University | In use | | 44 | 22 | Nov 2025 | Nov 2026 |
| Toolathlon-Verified 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. | HKUST NLP | Apps and MCP servers | University | In use | GLM-5.3-Flash 78.4% | 18 | 9 | Jun 2026 | Oct 2026 |
| Toolathlon Original Replaced in Jun 2026 | In use | | 24 | 12 | Oct 2025 | Jul 2026 | |||
| AA-LCR 1.1 Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key. | Artificial Analysis | Reasoning over long documents | University | In use | | 6 | 2 | Sep 2026 | Nov 2026 |
| AA-LCR (Artificial Analysis Long Context Reasoning) 1.0 Replaced in Sep 2026 | In use | | 36 | 18 | Aug 2025 | Oct 2026 | |||
| OmniDocBench v1.6 Models parse 1,651 PDF pages, including nested tables, dense formulas and unusual layouts, into text, tables and formulas. Scored by the v1.5 Overall formula. | Shanghai AI Lab (OpenDataLab) | Documents, charts and text in images | Everyday | In use | OXR-1.0 97.0 | 5 | 1 | Apr 2026 | Nov 2026 |
| OmniDocBench v1.5 Replaced in Apr 2026 | In use | GLM-OCR 94.6 | 34 | 14 | Sep 2025 | Oct 2026 | |||
| OmniDocBench 1.0 Replaced in Sep 2025 | Fading | 3 | 4 | Dec 2024 | May 2026 | ||||
| MBPP Models write short Python functions from a one-line task description. A solution counts as correct if it passes the task's three assert tests. | Google Research | Function-level problems | School | Fading | Sarvam 30B 92.7% | 41 | 55 | Aug 2021 | Oct 2026 |
| SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. | SciCode team | Scientific code | Expert | In use | | 41 | 26 | Jul 2024 | Oct 2026 |
| Agents' Last Exam v1.1 Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references. | UC Berkeley RDI (board hosted by Snorkel AI) | Tasks from real jobs | Expert | In use | GPT-6.2 Astra 47.5% | 8 | 2 | Oct 2026 | Nov 2026 |
| Agents' Last Exam 1.0 Replaced in Oct 2026 | UC Berkeley RDI + 300+ experts | In use | Claude Opus 5.5 38.2% | 32 | 8 | Jun 2026 | Nov 2026 | ||
| DROP Reading comprehension requiring discrete operations (counting, sorting, arithmetic) over paragraphs. Scored by F1. | AllenAI + UC Irvine | Reading comprehension | School | Fading | Kimi K2 93.5% | 36 | 39 | Mar 2019 | Sep 2026 |
| C-Eval 13,948 Chinese multiple-choice exam questions across 52 disciplines and four difficulty levels. Accuracy. | SJTU + Tsinghua + Edinburgh | Exam questions | University | Fading | | 36 | 26 | May 2023 | Sep 2026 |
| MultiChallenge Multi-turn conversations testing instruction retention, inference memory, versioned editing and self-coherence. Graded by LLM judge with rubrics. | Scale AI | Multi-turn conversation | Everyday | In use | GPT-6.1 Sol 82% | 36 | 21 | Jan 2025 | Nov 2026 |
| MRCR v2 Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. | Google DeepMind | Finding facts in long inputs | Everyday | In use | | 29 | 18 | Jun 2025 | Oct 2026 |
| MRCR (Michelangelo, original) v1 Replaced in Jun 2025 | Dropped | | 6 | 7 | Sep 2024 | Apr 2025 | |||
| MMStar Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. | USTC and Shanghai AI Laboratory | General visual questions | School | In use | | 35 | 22 | Mar 2024 | Oct 2026 |
| Codeforces rating Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. | Codeforces | Competitive programming | Competition | In use | | 34 | 35 | 2024 | Sep 2026 |
| MGSM-Rev2 Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy. | Reasoning in many languages | School | Not in cards | GPT-5 99.3% | Nov 2025 | Oct 2025 | |||
| MGSM v1 Replaced in Nov 2025 | Google Research | Fading | | 33 | 41 | Oct 2022 | Oct 2026 | ||
| IMO-AnswerBench 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. | Google DeepMind | Competition problems with answers | Competition | In use | GLM-5.2 91% | 33 | 20 | Nov 2025 | Oct 2026 |
| Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. | NTU S-Lab (LMMs-Lab) | Video understanding | University | In use | | 33 | 16 | Jan 2025 | Aug 2026 |
| MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. | University of Edinburgh | Exam questions | University | Not in cards | Aug 2024 | ||||
| MMLU-Redux 1.0 Replaced in Aug 2024 | Fading | | 32 | 25 | Jun 2024 | Jun 2026 | |||
| SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. | ByteDance Seed + M-A-P | Expert-written questions | Expert | In use | | 31 | 14 | Feb 2025 | Oct 2026 |
| ChartQA Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. | York University and Nanyang Technological University | Documents, charts and text in images | School | In use | | 30 | 23 | Mar 2022 | Aug 2026 |
| DocVQA Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers. | CVC, Universitat Autònoma de Barcelona and IIIT Hyderabad | Documents, charts and text in images | Everyday | In use | | 30 | 23 | Jul 2020 | Sep 2026 |
| LongBench v2 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. | Tsinghua KEG / Zhipu | Reasoning over long documents | Expert | In use | | 28 | 19 | Dec 2024 | Nov 2026 |
| LongBench v1 Replaced in Dec 2024 | THUDM (Tsinghua University) | University | Dropped | GLM-4 81.1% | 2 | Aug 2023 | Jul 2025 | ||
| LVBench Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. | Zhipu AI / Tsinghua | Long videos | Everyday | In use | | 30 | 19 | Jun 2024 | Nov 2026 |
| HellaSwag Commonsense sentence completion with adversarially filtered wrong endings. Multiple-choice accuracy. | University of Washington + AllenAI | Commonsense | Everyday | Fading | | 29 | 39 | May 2019 | Oct 2026 |
| Aider Polyglot Models edit code to solve 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust. Scored by the share passing unit tests. | Aider | Function-level problems | University | Fading | GPT-5 88% | 26 | 17 | Dec 2024 | Feb 2026 |
| Aider-Edit Edit (2023) Replaced in Dec 2024 | School | Dropped | | 3 | 4 | Jul 2023 | Oct 2025 | ||
| RealWorldQA Models answer a question about each of 700+ real-world photos, many taken from vehicles, that test basic spatial understanding. Answers are easy to verify. | xAI | Spatial reasoning and counting | Everyday | In use | | 29 | 19 | Apr 2024 | Sep 2026 |
| MMBench-EN v1.1 Models answer English multiple-choice image questions across 20 ability dimensions, using CircularEval with shuffled options. Scored by accuracy on the dev split. | Shanghai AI Laboratory (OpenCompass) | General visual questions | Everyday | In use | | 22 | 11 | Apr 2024 | Aug 2026 |
| MMBench 1.0 Replaced in Apr 2024 | Fading | | 6 | 11 | Jul 2023 | May 2026 | |||
| ERQA 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. | Google DeepMind | Embodied reasoning | Everyday | In use | | 28 | 15 | Mar 2025 | Oct 2026 |
| ScreenSpot-Pro GUI grounding on high-resolution professional-software screenshots. The model must locate the target UI element. | ScreenSpot-Pro authors | Finding things on screen | University | In use | GPT-6 Astra 92.7% | 28 | 15 | Jan 2025 | Nov 2026 |
| SimpleVQA Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. | Multimodal Art Projection (M-A-P) et al. | Short factual questions | University | In use | | 28 | 9 | Feb 2025 | Oct 2026 |
| NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. | ByteDance + M-A-P + 2077AI | Building whole projects | Expert | In use | | 28 | 6 | Dec 2025 | Nov 2026 |
| ZeroBench 100 hand-made visual questions that all frontier models originally failed. Scored by exact answer, pass@1 and pass@5. | University of Cambridge and others | General visual questions | University | In use | GPT-5.6 Sol 54.6% | 27 | 17 | Feb 2025 | Nov 2026 |
| ARC-Challenge Models answer grade-school science multiple-choice questions that retrieval and word co-occurrence baselines got wrong. Scored by accuracy. | Allen Institute for AI | Exam questions | School | Dropped | | 25 | 43 | Mar 2018 | Oct 2026 |
| WMT24++ English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. | Translation | University | In use | | 25 | 19 | Feb 2025 | Oct 2026 | |
| HallusionBench Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. | University of Maryland | Hallucination rates | Everyday | In use | | 25 | 18 | Oct 2023 | Oct 2026 |
| GraphWalks A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. | OpenAI | Reasoning over long documents | University | In use | Claude Fable 5 99.96% | 25 | 18 | Apr 2025 | Nov 2026 |
| BabyVision Model solves 388 perception-heavy visual puzzles (discrimination, tracking, spatial, patterns) that young children solve. Accuracy versus 94% human baseline. | UniPat AI + xbench | Spatial reasoning and counting | Everyday | In use | | 25 | 9 | Jan 2026 | Nov 2026 |
| RefCOCO / RefCOCO+ / RefCOCOg Model grounds a referring expression to a bounding box in a COCO image. Scored by accuracy at IoU 0.5. | UNC Chapel Hill (Licheng Yu et al.) | Detection and captioning | Everyday | In use | | 24 | 17 | Jul 2016 | Sep 2026 |
| Artificial Analysis Intelligence Index v5 5.0 Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. | Artificial Analysis | Composite indices | In use | Claude Fable 5.2 64.6 | 7 | 3 | Oct 2026 | Nov 2026 | |
| Artificial Analysis Intelligence Index v4.3 Replaced in Oct 2026 | In use | | 4 | 2 | Sep 2026 | Oct 2026 | |||
| Artificial Analysis Intelligence Index v4.2 Replaced in Sep 2026 | Not in cards | Sep 2026 | |||||||
| Artificial Analysis Intelligence Index v4.1 Replaced in Sep 2026 | In use | Grok 4.6 61.0 | 7 | 3 | Jun 2026 | Aug 2026 | |||
| Artificial Analysis Intelligence Index v4.0 Replaced in Jun 2026 | Fading | GPT-5.5 60.2 | 3 | 3 | Jan 2026 | May 2026 | |||
| Artificial Analysis Intelligence Index v3 Replaced in Jan 2026 | Dropped | | 2 | Sep 2025 | Oct 2025 | ||||
| Artificial Analysis Intelligence Index v2.2 Replaced in Sep 2025 | Not in cards | Aug 2025 | |||||||
| Artificial Analysis Intelligence Index v2 Replaced in Aug 2025 | Dropped | GLM-4.5 67.7 | 1 | 2 | Feb 2025 | Aug 2025 | |||
| Artificial Analysis Intelligence Index v1 Replaced in Feb 2025 | Not in cards | Jan 2024 | |||||||
| FrontierCode 2.0 Agents resolve 132 maintainer-built tasks from deliberately ambiguous prompts. Rubric checks ask whether a maintainer would merge the change. | Cognition | Fixing real repositories | Expert | In use | Claude Fable 5.2 50.2% | 10 | 4 | Oct 2026 | Nov 2026 |
| FrontierCode 1.1 Replaced in Oct 2026 | In use | Claude Fable 5 53.5% | 10 | 7 | Jul 2026 | Nov 2026 | |||
| FrontierCode 1.0 Replaced in Jul 2026 | In use | Claude Opus 4.8 34.3% | 4 | 2 | Jun 2026 | Sep 2026 | |||
| HMMT Nov 2025 Models solve the 30 problems of the November 2025 Harvard-MIT Mathematics Tournament. Final answers are checked automatically against the answer key. | MathArena (ETH Zurich SRI Lab), problems by HMMT | Competition problems with answers | Competition | Fading | GPT-5.2 99.2% | 24 | 11 | Nov 2025 | Jun 2026 |
| WinoGrande 44K Winograd-style pronoun-resolution problems. Binary-choice accuracy. | AllenAI | Commonsense | Everyday | Fading | | 22 | 41 | Jul 2019 | Oct 2026 |
| MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. | Yale NLP | Video understanding | University | In use | | 22 | 14 | Jan 2025 | Aug 2026 |
| AndroidWorld Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. | Google Research | Phones | Everyday | In use | GPT-6 Astra 100% | 22 | 9 | May 2024 | Oct 2026 |
| HumanEval+ Models write Python functions for the 164 HumanEval problems. EvalPlus checks them against about 80 times more tests than the original. | UIUC (EvalPlus) | Function-level problems | School | Fading | | 21 | 18 | May 2023 | Apr 2026 |
| CountBench Models count the objects in natural images from the CountBench image-text set. Lab cards often use a question-answer version, CountBenchQA, scored by accuracy. | Google Research | Spatial reasoning and counting | Everyday | In use | | 21 | 15 | Feb 2023 | Aug 2026 |
| MMLU-ProX MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. | University of Tokyo (Li Lab) and collaborators | Knowledge in many languages | University | In use | | 21 | 8 | Mar 2025 | Oct 2026 |
| MLVU Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. | BAAI + partners | Long videos | Everyday | In use | | 21 | 8 | Jun 2024 | Aug 2026 |
| Multi-IF Models follow verifiable instructions over three-turn conversations in eight languages, extending IFEval. Scored by instruction accuracy at each turn. | Meta | Multi-turn conversation | Everyday | In use | LFM3-1.7B 89.1% | 20 | 15 | Oct 2024 | Oct 2026 |
| τ³-bench Banking Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. | Sierra | Customer service | Everyday | In use | | 20 | 11 | Mar 2026 | Nov 2026 |
| MedXpertQA-MM Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. | Tsinghua University (TsinghuaC3I) | Medicine | Expert | In use | | 20 | 7 | Jan 2025 | Oct 2026 |
| TriviaQA Trivia questions with evidence documents. Scored by exact match. | University of Washington | Trivia and open questions | Everyday | Fading | | 19 | 32 | May 2017 | Oct 2026 |
| BLINK Models answer 3,807 multiple choice questions built from 14 classic computer vision tasks that use one or more images. Scoring is accuracy. | Xingyu Fu et al. | General visual questions | Everyday | In use | | 19 | 24 | Apr 2024 | Oct 2026 |
| Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. | Vals AI | Finance | Expert | In use | Claude Fable 5.2 69.5% | 1 | 1 | Oct 2026 | Nov 2026 |
| Vals Finance Agent v2 Replaced in Oct 2026 | In use | | 10 | 9 | May 2026 | Oct 2026 | |||
| Vals Finance Agent v1.1 Replaced in May 2026 | Fading | Claude Opus 4.7 64.4% | 7 | 5 | Feb 2026 | Jun 2026 | |||
| Vals Finance Agent v1 Replaced in Feb 2026 | Vals AI (with Stanford researchers and a global systemically important bank) | Dropped | Claude Opus 4.6 60.6% | 1 | 1 | Apr 2025 | Feb 2026 | ||
| INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. | EPFL and Cohere For AI | Knowledge in many languages | University | Fading | Claude Opus 5 89.8% | 19 | 14 | Nov 2024 | Jul 2026 |
| OpenAI MRCR v2 Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. | OpenAI | Finding facts in long inputs | Everyday | In use | | 13 | 6 | Dec 2025 | Nov 2026 |
| OpenAI MRCR v1 Replaced in Dec 2025 | Dropped | | 6 | 7 | Apr 2025 | Aug 2026 | |||
| MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. | Shanghai AI Laboratory (OpenGVLab) | Video understanding | Everyday | Fading | | 19 | 12 | Nov 2023 | Aug 2026 |
| BrowseComp-ZH Agents browse the Chinese web to answer 289 multi-hop questions with short, verifiable answers. Scored by accuracy. | Researchers from Peking University, Mindverse AI, and other groups | Deep research | Expert | Fading | | 19 | 8 | Apr 2025 | Mar 2026 |
| CyberGym (final-submission metric) final-submission (2026-07) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. | UC Berkeley (Sunblaze) | Finding and exploiting real bugs | Expert | In use | | 1 | Jul 2026 | Jul 2026 | |
| CyberGym any-of (2025) Replaced in Jul 2026 | In use | Grok 4.6 79.7% | 17 | 23 | Jun 2025 | Oct 2026 | |||
| ProgramBench 200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original. | Meta Superintelligence Labs, Stanford University and Harvard University | Building whole projects | Research frontier | In use | Claude Opus 5.5 18.5% | 18 | 15 | May 2026 | Nov 2026 |
| MMLongBench-Doc Models answer 1,062 questions about 130 long PDF documents averaging 49 pages, and some questions are unanswerable. Scoring is accuracy and F1. | Yubo Ma et al. | Documents, charts and text in images | University | Fading | | 18 | 10 | Jul 2024 | Oct 2026 |
| CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. | MBZUAI + others (Li et al.) | Exam questions | University | Fading | | 18 | 8 | Jun 2023 | Oct 2026 |
| JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. | University of Washington + collaborators | Tasks from real jobs | Expert | In use | GLM-5.3 61.4% | 18 | 6 | May 2026 | Nov 2026 |
| AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. | Microsoft | Exam questions | School | Fading | | 17 | 39 | Apr 2023 | Oct 2026 |
| HealthBench Professional 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. | OpenAI | Medicine | Expert | In use | Claude Sonnet 5.5 69.2% | 17 | 34 | Apr 2026 | Nov 2026 |
| ScreenSpot-v2 Models locate a described element on mobile, desktop and web screenshots. A click inside the target box counts as correct. | OS-Atlas team (Shanghai AI Lab and others) | Finding things on screen | Everyday | Fading | | 9 | 12 | Oct 2024 | Oct 2026 |
| ScreenSpot v1 Replaced in Oct 2024 | Nanjing University (SeeClick) | Fading | | 8 | 9 | Jan 2024 | Apr 2026 | ||
| AA-Omniscience v1.1 Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. | Artificial Analysis | Short factual questions | Expert | In use | | 2 | 1 | Oct 2026 | Nov 2026 |
| AA-Omniscience 1.0 Replaced in Oct 2026 | In use | | 15 | 18 | Nov 2025 | Oct 2026 | |||
| DynaMath Models answer visual math questions generated from 501 seed programs that vary values and graphs. Scored by average and worst-case accuracy across variants. | University of Illinois Urbana-Champaign and UC Berkeley | Visual math and diagrams | School | Fading | | 17 | 12 | Oct 2024 | Jun 2026 |
| PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. | Qwen Team, Alibaba Group | Competition problems with answers | Competition | In use | | 17 | 11 | Apr 2025 | Aug 2026 |
| WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. | ByteDance Seed | Questions that need search | Everyday | In use | | 17 | 6 | Aug 2025 | Oct 2026 |
| VitaBench 2.0 Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics. | Meituan LongCat Team | Customer service | Everyday | In use | Claude Opus 4.6 50.3% | 1 | May 2026 | Oct 2026 | |
| VITA-Bench 1.0 Replaced in May 2026 | Meituan LongCat team | Fading | | 16 | 5 | Sep 2025 | Jun 2026 | ||
| RULERv2 Models solve synthetic multi-key and multi-value needle tasks and QA tasks at basic, easy, medium and hard levels. Contexts run from 8K to 1M tokens, and accuracy is averaged across lengths. | NVIDIA | Finding facts in long inputs | Everyday | Not in cards | GPT-4.1 89.2% | Oct 2025 | Jul 2025 | ||
| RULER v1 Replaced in Oct 2025 | In use | | 16 | 37 | Apr 2024 | Oct 2026 | |||
| LiveBench Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. | Abacus.AI, NYU, NVIDIA, et al. | Other | University | Dropped | | 16 | 23 | Jun 2024 | Nov 2026 |
| ZebraLogic Models solve logic grid puzzles of controlled size, derived from constraint satisfaction problems. Scored by the share of puzzles solved fully. | Allen Institute for AI | Abstract puzzles and logic | School | Fading | | 16 | 19 | Feb 2025 | Feb 2026 |
| MT-Bench 80 two-turn questions graded by GPT-4 as judge. Introduced LLM-as-judge methodology. | LMSYS | Multi-turn conversation | Everyday | Fading | | 16 | 16 | Jun 2023 | Apr 2026 |
| DeepSearchQA Questions that need multi-step search and synthesis. Final answers graded against references. | Google DeepMind | Deep research | Expert | In use | Kimi K3 95% | 16 | 13 | Jan 2026 | Nov 2026 |
| Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. | Surge AI | Documents, charts and text in images | Expert | In use | GPT-6.2 Astra 90.7% | 16 | 11 | Aug 2026 | Nov 2026 |
| AA-Briefcase v1.1 Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. | Artificial Analysis | Tasks from real jobs | Expert | In use | Claude Opus 5.5 1,822 | 10 | 5 | Sep 2026 | Nov 2026 |
| AA-Briefcase v1 Replaced in Sep 2026 | In use | Claude Opus 5 1,720 | 6 | 6 | 2026 | Oct 2026 | |||
| PIQA Models pick which of two solutions better achieves an everyday physical goal. Scoring is accuracy. | Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi | Commonsense | Everyday | Dropped | | 15 | 32 | Nov 2019 | Oct 2026 |
| InfographicVQA Models answer about 30,000 questions on about 5,000 infographics that mix text, layout, charts, and graphics. Scoring is ANLS. | Computer Vision Center (UAB) and IIIT Hyderabad | Documents, charts and text in images | School | In use | Kimi K2.5 92.6% | 15 | 18 | Apr 2021 | Aug 2026 |
| Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. | Cohere Labs | Knowledge in many languages | University | In use | Claude Opus 5 95.7% | 3 | 4 | May 2026 | Oct 2026 |
| Global MMLU Lite 2.0 Replaced in May 2026 | System cards only | | 1 | Dec 2025 | Dec 2025 | ||||
| Global MMLU Lite 1.0 Replaced in Dec 2025 | Dropped | Claude Opus 5 95.7% | 12 | 12 | Dec 2024 | Oct 2026 | |||
| FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. | Speech recognition and translation | Everyday | In use | | 15 | 14 | May 2022 | Oct 2026 | |
| GDP.pdf 100 expert multimodal reasoning tasks over real economically important PDF documents. Scored by rubric-criteria pass rate. | Surge AI | Documents, charts and text in images | Expert | In use | | 15 | 10 | Jul 2026 | Nov 2026 |
| AlpacaEval 2.0 Models answer 805 open instructions, and a GPT-4 Turbo judge compares each reply with GPT-4 Turbo's own. The score is a win rate. | Tatsu Lab, Stanford University | Overall chat quality | Everyday | Fading | | 15 | 9 | Jan 2024 | Apr 2026 |
| AlpacaEval 1.0 Replaced in Jan 2024 | Stanford Tatsu Lab | Not in cards | GPT-4 Turbo 97.7% | May 2023 | Dec 2023 | ||||
| Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. | Stanford-led community (Terminal-Bench team + domain experts) | Research science | Expert | In use | GPT-6 Astra 68.1% | 15 | 9 | Aug 2026 | Nov 2026 |
| V* Models answer 191 multiple choice questions about small details in high-resolution images, such as an object's color or relative position. | Penghao Wu and Saining Xie | General visual questions | Everyday | Fading | | 15 | 5 | Dec 2023 | Oct 2026 |
| CC-OCR-Bench-V2 Models process 7,093 hard real-world document samples across five OCR tracks, including parsing and document QA. Each track uses its own metric. | Chunyi Peng et al. (with Qwen team members) | Documents, charts and text in images | Everyday | In use | | 1 | May 2026 | Aug 2026 | |
| CC-OCR v1 Replaced in May 2026 | Alibaba Tongyi Lab (OCR team) | Fading | | 14 | 4 | Dec 2024 | Apr 2026 | ||
| FrontierMath Tiers 1-3 v2 Models solve 295 unpublished math problems from undergraduate to advanced graduate level and submit answers that code checks automatically. Scored by share correct. | Epoch AI | Research mathematics | Expert | In use | Claude Fable 5.2 95.1% | 3 | Jun 2026 | Nov 2026 | |
| FrontierMath Original Replaced in Jun 2026 | Fading | GPT-5.5 Pro 52.4% | 12 | 1 | Nov 2024 | Jun 2026 | |||
| MBPP+ Models write short Python functions for MBPP problems, and EvalPlus checks each one against many more generated tests than the original. | UIUC (EvalPlus) | Function-level problems | School | Fading | | 14 | 10 | Dec 2023 | Apr 2026 |
| CritPt v2 Models solve unpublished research-level physics challenges written by working physicists, with answers graded automatically. The v2 release revised the public test challenges. | CritPt collaboration | Research science | Research frontier | In use | GPT-6 Astra 83.1% | 2 | 1 | Oct 2026 | Nov 2026 |
| CritPt v1 Replaced in Oct 2026 | In use | GPT-6 Astra 83.1% | 12 | 6 | Sep 2025 | Oct 2026 | |||
| OJBench Models write C++ or Python solutions to 232 competition problems from NOI and ICPC. Judge test cases score each solution by pass@1. | Beijing University of Posts and Telecommunications, Tsinghua University, and others | Competitive programming | Competition | Fading | | 14 | 7 | Jun 2025 | Jun 2026 |
| EmbSpatialBench Models answer multiple-choice questions on six spatial relations between objects in egocentric embodied scenes. Scored by accuracy. | Fudan University | Spatial reasoning and counting | Everyday | Fading | | 14 | 5 | Jun 2024 | Jun 2026 |
| SWE Atlas v1.1 Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. | Scale AI | Fixing real repositories | Expert | Not in cards | Claude Fable 5.2 80.5% | Nov 2026 | Nov 2026 | ||
| SWE Atlas Original Replaced in Nov 2026 | In use | Claude Opus 5 63.2% | 14 | 2 | Mar 2026 | Nov 2026 | |||
| Global PIQA Models pick the correct solution to everyday commonsense prompts in over 100 languages, many about local foods and customs. Scoring is accuracy. | MRL Benchmarks (over 350 researchers) | Reasoning in many languages | Everyday | Fading | | 14 | 1 | Oct 2025 | Jun 2026 |
| Claw-Eval 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. | Claw-Eval team (academic) | General assistants | Everyday | In use | Claude Sonnet 4.6 81.4% | 14 | 1 | Mar 2026 | Oct 2026 |
| TruthfulQA Models answer 817 questions across 38 categories that are written to elicit common misconceptions. Scoring checks for truthful answers, often in multiple-choice form. | University of Oxford and OpenAI | Short factual questions | Everyday | Fading | | 13 | 28 | Sep 2021 | Oct 2026 |
| Beyond AIME Models answer 100 original competition-style math problems at or above the hardest AIME level. Each answer is a positive integer checked by exact match. | ByteDance Seed | Competition problems with answers | Competition | Fading | | 13 | 8 | Apr 2025 | Jun 2026 |
| PinchBench 2.0 Models drive an OpenClaw agent through 148 real-world tasks, including data and log analysis, meeting notes, DevOps, PDFs, research and writing. Graded by checks and a Haiku judge. | Kilo AI | General assistants | Everyday | Not in cards | Claude Opus 4.8 93.5% | May 2026 | Oct 2026 | ||
| PinchBench 1.x Replaced in May 2026 | In use | | 13 | 6 | Feb 2026 | Oct 2026 | |||
| VLMs Are Blind Models answer seven simple visual tasks such as counting line crossings or overlapping circles, which young children solve easily. Scored by accuracy. | Auburn University and University of Alberta | Spatial reasoning and counting | Everyday | Fading | | 13 | 5 | Jul 2024 | Apr 2026 |
| SkillsBench 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. | BenchFlow + 105 domain experts | Terminal tasks | Expert | In use | GPT-5.5 66.5% | 13 | 2 | Feb 2026 | Nov 2026 |
| MMAU v05.15.25 Models answer multiple-choice questions about speech, sound and music clips that need expert knowledge and reasoning. Scored by accuracy on test-mini or the hidden-answer test set. | University of Maryland (GAMMA lab) | Audio understanding and voice | University | Not in cards | | May 2025 | Dec 2025 | ||
| MMAU Original Replaced in May 2025 | University of Maryland | In use | | 12 | 14 | Oct 2024 | Sep 2026 | ||
| Harvey LAB v1.1 (HLAB-Vals-1.1) Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes. | Harvey (board run by Vals AI) | Law | Expert | In use | | 2 | Oct 2026 | Nov 2026 | |
| Harvey LAB (Legal Agent Benchmark) 1.0 Replaced in Oct 2026 | Harvey | In use | | 10 | 11 | May 2026 | Oct 2026 | ||
| WebVoyager Agents complete user tasks on 15 popular real websites by browsing from screenshots and page text. A GPT-4V judge grades whether each task succeeded. | WebVoyager team (He et al., ACL 2024) | Web browsing | Everyday | Fading | | 12 | 8 | Jan 2024 | Oct 2026 |
| MathArena (Apex, IMO, USAMO) Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. | ETH Zurich SRI Lab | Competition problems with answers | Competition | In use | Claude Opus 4.8 81.3% | 12 | 6 | Feb 2025 | Nov 2026 |
| RefSpatial-Expand-Bench Models point to an object or free space described by multi-step spatial relations, in more indoor scenes plus new outdoor streets and parking lots. Scored by mask hit. | BAAI (RoboRefer) | Embodied reasoning | Everyday | Not in cards | Oct 2025 | ||||
| RefSpatial-Bench 1.0 Replaced in Oct 2025 | Fading | | 12 | 6 | Jun 2025 | May 2026 | |||
| COLLIE Models generate text that meets compositional constraints at word, sentence, paragraph, and passage level. Scored by the share passing automatic constraint checks. | Princeton NLP | Following precise instructions | Everyday | Fading | GPT-5 99% | 12 | 5 | Jul 2023 | May 2026 |
| EgoSchema Models answer five-way multiple-choice questions about three-minute egocentric video clips from Ego4D. Scored by accuracy on 5,031 questions. | UC Berkeley | Long videos | Everyday | Dropped | | 12 | 5 | Aug 2023 | Apr 2025 |
| ODinW-13 Models detect objects in 13 varied real-world datasets, such as aquarium and aerial images, from text category names. Scored by mean average precision. | Microsoft Research | Detection and captioning | Everyday | Fading | | 12 | 4 | Dec 2021 | Jun 2026 |
| Claw-Eval Pass^3 Agents complete 300 human-verified tasks in service orchestration, multimodal interaction, and professional dialogue. Pass^3 credits a task only if all three trials pass. | Claw-Eval team | General assistants | University | In use | | 12 | 2 | Apr 2026 | Nov 2026 |
| BoolQ Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy. | Reading comprehension | Everyday | Dropped | PaLM 2-L 90.9% | 11 | 25 | May 2019 | Jun 2025 | |
| ARC-Easy Models answer grade-school science multiple-choice questions from the ARC Easy Set, which has 5,197 questions. Scoring is accuracy. | Allen Institute for AI | Exam questions | School | Dropped | | 11 | 20 | Mar 2018 | Jun 2025 |
| MultiPL-E Models write code for HumanEval and MBPP problems translated into 18 other programming languages. Scored by pass@1 on unit tests. | Northeastern University Programming Research Lab (NuPRL) et al. | Function-level problems | School | Fading | | 11 | 19 | Aug 2022 | Apr 2026 |
| VoiceBench Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. | CUHK-Shenzhen | Audio understanding and voice | Everyday | In use | | 11 | 12 | Oct 2024 | Oct 2026 |
| SWE-bench Multimodal v2 Agents patch JavaScript repositories using issue reports that include images. Automated tests check whether each patch fixes the issue. | SWE-bench team | Fixing real repositories | Expert | System cards only | Claude Opus 5.5 70.2% | 1 | Sep 2026 | Nov 2026 | |
| SWE-bench Multimodal Original Replaced in Sep 2026 | SWE-bench team (Princeton/Stanford) | In use | Claude Fable 5.2 74.1% | 11 | 10 | Oct 2024 | Nov 2026 | ||
| TextVQA Models answer 45,336 questions about 28,408 images that require reading text in the image. Scoring is VQA accuracy against human answers. | Facebook AI Research | Documents, charts and text in images | Everyday | In use | | 11 | 11 | Apr 2019 | Oct 2026 |
| CoVoST2 Models translate speech from 21 languages into English and from English into 15 languages. Scored by BLEU. | Meta AI (Facebook AI) | Speech recognition and translation | University | Fading | | 11 | 11 | Jul 2020 | Oct 2026 |
| CursorBench 4.1 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported. | Cursor | Fixing real repositories | Expert | In use | | 1 | 1 | Nov 2026 | Nov 2026 |
| CursorBench 4.0 Replaced in Nov 2026 | In use | | 3 | 4 | Sep 2026 | Nov 2026 | |||
| CursorBench 3.2 Replaced in Sep 2026 | In use | Claude Fable 5.1 73.4% | 2 | 2 | Jul 2026 | Sep 2026 | |||
| CursorBench 3.1 Replaced in Jul 2026 | Fading | Claude Fable 5 72.9% | 3 | 2 | May 2026 | Jun 2026 | |||
| CursorBench 3.0 Replaced in May 2026 | System cards only | Composer 2 61.3% | 1 | Mar 2026 | Mar 2026 | ||||
| Cursor Bench (2025 internal version) 2025 (before 3.0) Replaced in Mar 2026 | Fading | 2 | Oct 2025 | Feb 2026 | |||||
| Chinese SimpleQA 3,000 short Chinese factual questions across six topics. Graded correct, incorrect or not attempted. | Alibaba (Taobao & Tmall) | Short factual questions | University | Fading | | 11 | 10 | Nov 2024 | Jun 2026 |
| APEX-Agents 1.1 Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero. | Mercor | Simulated companies | Expert | Not in cards | | Sep 2026 | Nov 2026 | ||
| APEX-Agents 1.0 Replaced in Sep 2026 | In use | Kimi K3.1 Pro 67.2% | 11 | 5 | Jan 2026 | Nov 2026 | |||
| GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. | OpenAI | Tasks from real jobs | Expert | In use | | 11 | 2 | Sep 2025 | Sep 2026 |
| NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. | Peking University and Alibaba Group | Knowledge in many languages | University | Fading | | 11 | Nov 2025 | Jun 2026 | |
| MAXIFE Models follow 1,667 verifiable instructions in 23 languages and in cross-lingual settings. Rule-based checks and model judges score compliance. | OPPO (Mente Lab) | Chat and instructions in many languages | Everyday | Fading | | 11 | Jun 2025 | Jun 2026 | |
| HealthBench 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. | OpenAI | Medicine | Expert | In use | GPT-5 67.2% | 10 | 36 | May 2025 | Nov 2026 |
| PostTrainBench 1.2 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | AISA group (ELLIS Institute Tübingen and MPI-IS) + University of Tübingen + Thoughtful Lab | Machine learning engineering | Research frontier | In use | Claude Fable 5.1 44.6% | 3 | Sep 2026 | Nov 2026 | |
| PostTrainBench 1.1 Replaced in Sep 2026 | In use | Claude Fable 5 41.8% | 2 | Jul 2026 | Oct 2026 | ||||
| PostTrainBench 1.0 Replaced in Jul 2026 | In use | Claude Opus 4.8 34.1% | 5 | 13 | Mar 2026 | Nov 2026 | |||
| FACTS Grounding v2 Models write long answers from a supplied document of up to 32k tokens. Judge models check that each answer stays faithful to the document. | Google DeepMind + Google Research + Kaggle | Faithful to sources | University | Not in cards | GPT-6.1 Terra 84.2% | Dec 2025 | Oct 2026 | ||
| FACTS Grounding v1 Replaced in Dec 2025 | Google DeepMind + Kaggle | Dropped | Kimi K2 88.5% | 10 | 11 | Jan 2025 | Oct 2026 | ||
| WritingBench Models complete 1,000 real-world writing requests across 6 domains and 100 subdomains. A judge model scores each reply on five query-specific criteria. | Alibaba Group (X-PLUG) | Creative writing | University | Fading | GPT-5 83.9 | 10 | 11 | Mar 2025 | Jul 2026 |
| We-Math Models solve visual math problems decomposed into sub-problems by knowledge concept. Scoring compares sub-problem and full-problem results to diagnose knowledge and generalization gaps. | Beijing University of Posts and Telecommunications | Visual math and diagrams | School | Fading | | 10 | 9 | Jul 2024 | Jun 2026 |
| SWE-Lancer Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned. | OpenAI | Fixing real repositories | Expert | Fading | GPT-5.3 Codex 81.4% | 10 | 9 | Feb 2025 | Feb 2026 |
| SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials. | Abundant AI | Building whole projects | Expert | In use | | 1 | Jul 2026 | Nov 2026 | |
| SWE-Marathon 1.0 Replaced in Jul 2026 | In use | Grok 4.5 29% | 9 | 6 | Jun 2026 | Oct 2026 | |||
| Online-Mind2Web Web agents do 300 tasks on 136 live websites. An LLM judge reads the agent's trajectory and decides whether the task succeeded. | Ohio State University (OSU NLP Group) | Web browsing | Everyday | Fading | | 10 | 6 | Apr 2025 | Jun 2026 |
| MCPMark Verified Agents complete 127 tasks through version-pinned MCP servers. Stabilized verifiers score single-run task success. | eval-sys (MCPMark team) | Apps and MCP servers | University | Not in cards | Kimi K3 96.1% | Jun 2026 | Jul 2026 | ||
| MCPMark Original Replaced in Jun 2026 | In use | Kimi K3 94.5% | 10 | 3 | Sep 2025 | Jul 2026 | |||
| PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. | Shanghai Jiao Tong University and Shanghai AI Laboratory | Medicine | University | Fading | | 10 | 1 | May 2023 | Aug 2026 |
| SLAKE Models answer English and Chinese questions about radiology images of the head, neck, chest, abdomen, and pelvis. Scoring is accuracy. | Hong Kong Polytechnic University (Xiao-Ming Wu group) | Medicine | University | Fading | | 10 | 1 | Feb 2021 | Aug 2026 |
| DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. | Qwen Team, Alibaba Group | General assistants | Everyday | Fading | Claude Opus 4.6 58.9% | 10 | Jan 2026 | Jun 2026 | |
| OpenBookQA Models answer elementary science multiple-choice questions that need a given core science fact plus common knowledge. Scored by accuracy. | Allen Institute for AI | Exam questions | School | Fading | Hermes 4.3 36B 96.6% | 9 | 22 | Sep 2018 | Oct 2026 |
| ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. | UC Berkeley (Dawn Song lab) | Finding and exploiting real bugs | Expert | In use | GPT-6 Astra 42.4% | 9 | 22 | May 2026 | Nov 2026 |
| BBEH micro average Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. | Google DeepMind | Language reasoning | University | In use | | 9 | 13 | Feb 2025 | Sep 2026 |
| MuirBench Models answer multiple choice questions over sets of several images, including unanswerable variants. Scored by accuracy. | Fei Wang, Xingyu Fu et al. | General visual questions | Everyday | In use | | 9 | 12 | Jun 2024 | Nov 2026 |
| Terminal-Bench Hard Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. | Artificial Analysis | Terminal tasks | Expert | In use | | 9 | 9 | Sep 2025 | Jul 2026 |
| IMO 2026 Models write full proofs for the six problems of the 2026 International Mathematical Olympiad. Proofs are scored out of 42 points. | International Mathematical Olympiad, hosted in Shanghai, China | Written proofs | Competition | System cards only | Claude Opus 5 100% | 1 | Jul 2026 | Jul 2026 | |
| IMO 2025 Replaced in Jul 2026 | International Mathematical Olympiad, hosted by the Australian Maths Trust | Fading | | 9 | 7 | Jul 2025 | Jun 2026 | ||
| Common Voice Crowd-sourced multilingual read-speech corpus. Used here as an ASR test set scored by word error rate. | Mozilla | Speech recognition and translation | Everyday | Fading | | 9 | 8 | Dec 2019 | Jul 2026 |
| Multi-SWE-bench Agents resolve 1,632 GitHub issues in Java, TypeScript, JavaScript, Go, Rust, C, and C++ repos. Patches must pass the repos' tests. | ByteDance Seed | Fixing real repositories | Expert | Fading | | 9 | 6 | Apr 2025 | May 2026 |
| Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. | Apple (dataset), Meta AI (Omni3D split) | Detection and captioning | Everyday | Fading | | 9 | 1 | Nov 2020 | Mar 2026 |
| FrontierMath Tier 4 v2 Models solve 43 unpublished research-level math problems and submit answers that code checks automatically. Scored by share correct. | Epoch AI | Research mathematics | Research frontier | In use | GPT-6.1 Sol 100% | 3 | Jun 2026 | Nov 2026 | |
| FrontierMath Tier 4 Original Replaced in Jun 2026 | Fading | GPT-5.5 Pro 39.6% | 6 | Jul 2025 | Sep 2026 | ||||
| ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. | Carnegie Mellon (Lee & Brumley) | Finding and exploiting real bugs | Expert | In use | GPT-6 Astra 100% | 8 | 28 | May 2026 | Nov 2026 |
| SIQA Models answer multiple-choice questions about people's motives, reactions, and emotions in everyday social situations. Scored by accuracy. | Allen Institute for AI and University of Washington | Commonsense | Everyday | Dropped | | 8 | 23 | Apr 2019 | Oct 2026 |
| Natural Questions Models answer real Google search questions. Model cards usually use the open-domain version, scored by exact match against short answers from Wikipedia. | Trivia and open questions | Everyday | Dropped | | 8 | 18 | Jan 2019 | Aug 2025 | |
| FrontierSWE v2 34 ultra-long-horizon engineering and research tasks. 5 trials of up to 20 hours each. Score is mean task score 0-100. | Proximal | Building whole projects | Research frontier | In use | Claude Fable 5.2 69.9% | 4 | 9 | Sep 2026 | Nov 2026 |
| FrontierSWE v1 Replaced in Sep 2026 | In use | Claude Fable 5 88.2% | 4 | 4 | Apr 2026 | Aug 2026 | |||
| Creative Writing v3 Models write responses to 32 creative writing prompts, three times each. A judge model scores them with a rubric and pairwise Elo comparisons. | EQ-Bench | Creative writing | University | Dropped | Kimi K3.1 Pro 83.2 | 8 | 11 | Mar 2025 | Oct 2026 |
| BioMysteryBench Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. | Anthropic | Lab and data work | Expert | In use | Claude Fable 5.2 83.7% | 8 | 11 | Feb 2026 | Nov 2026 |
| LongVideoBench Models answer referring reasoning questions over 3,763 web videos up to an hour long, with interleaved subtitles. Scored by multiple-choice accuracy. | NTU S-Lab et al. | Long videos | Everyday | Fading | | 8 | 10 | Jul 2024 | Oct 2026 |
| BigCodeBench 1,140 practical Python tasks requiring calls to 139 libraries. Scored pass@1 on unit tests. | BigCode project | Function-level problems | University | Fading | | 8 | 9 | Jun 2024 | Oct 2026 |
| MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. | Zhipu AI and Tsinghua University (THUDM) | Video understanding | Everyday | Fading | | 8 | 8 | Jan 2025 | Oct 2026 |
| LibriSpeech Models transcribe English audiobook speech. Scored by word error rate on the test-clean and test-other splits. | Panayotov et al. (Johns Hopkins University) | Speech recognition and translation | Everyday | Fading | | 8 | 8 | Apr 2015 | Apr 2026 |
| MM-Vet Models answer open questions about images that combine recognition, OCR, knowledge, spatial reasoning, language generation and math. GPT-4 grades each answer against a reference. | Weihao Yu, Zhengyuan Yang et al. (NUS and Microsoft) | General visual questions | Everyday | Dropped | | 8 | 7 | Aug 2023 | Mar 2026 |
| Vibe-Eval Models answer 269 image-based prompts, 100 of them hard, that have expert gold responses. A Reka Core judge rates each answer. | Reka AI | General visual questions | Everyday | Dropped | | 8 | 6 | May 2024 | Feb 2026 |
| LiveCodeBench Pro Continuously updated Codeforces, ICPC and IOI problems annotated by olympiad medalists. Reported as Elo. | LiveCodeBench Pro team | Competitive programming | Competition | Fading | o4-mini 2,116 | 8 | 5 | Jun 2025 | Oct 2026 |
| WorldVQA Model names visual entities in 3,500 image-question pairs across nine categories, isolating memorized visual world knowledge. Exact-answer accuracy. | Moonshot AI | General visual questions | University | In use | | 8 | 4 | Jan 2026 | Oct 2026 |
| Claw-Eval-MM Agents complete the 101 multimodal tasks of Claw-Eval, such as video questions, document extraction, and webpage generation. Rubric items graded from run traces set scores. | Claw-Eval team | General visual questions | University | In use | Claude Opus 4.6 54.7% | 8 | 2 | Apr 2026 | Sep 2026 |
| LingoQA Models answer free-form questions about short driving videos. A trained classifier, Lingo-Judge, scores whether each answer is truthful. | Wayve | General visual questions | Everyday | In use | | 8 | 2 | Dec 2023 | Aug 2026 |
| FrontierScience Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. | OpenAI | Research science | Expert | In use | | 8 | 1 | Dec 2025 | Nov 2026 |
| Nuscene Models localize named objects as 3D boxes in nuScenes driving camera images. Scored by mAP. | Motional (nuTonomy) | Detection and captioning | Everyday | Fading | | 8 | Mar 2019 | Mar 2026 | |
| HealthBench Hard Models answer the 1,000 hardest HealthBench conversations, graded against physician-written rubrics. | OpenAI | Medicine | Expert | Fading | GPT-5 46.2% | 7 | 28 | May 2025 | Oct 2026 |
| SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. | University of Illinois Urbana-Champaign (Lingming Zhang) and University of Melbourne | Finding and exploiting real bugs | Expert | Not in cards | GPT-5.5 58.4% | Jun 2026 | Apr 2026 | ||
| SEC-bench Pro v260505 Replaced in Jun 2026 | University of Illinois Urbana-Champaign (Lingming Zhang) | In use | | 7 | 21 | May 2026 | Nov 2026 | ||
| LogicVista Models answer 448 multiple-choice questions on spatial, deductive, inductive, numeric, and mechanical reasoning set in images. Scored by accuracy. | UCLA | Visual math and diagrams | School | In use | | 7 | 12 | Jul 2024 | Aug 2026 |
| MathVerse Models solve diagram-based math problems shown in six versions that move information between the text and the image. It tests whether models read the diagram. | CUHK MMLab and Shanghai AI Laboratory | Visual math and diagrams | School | Fading | | 7 | 11 | Mar 2024 | Oct 2026 |
| Bird-SQL Models convert natural language questions into SQL over 95 large real databases across 37 domains. Scored by execution accuracy. | HKU and Alibaba DAMO Academy | Data science and SQL | University | Fading | GPT-5.5 72.5% | 7 | 10 | May 2023 | Oct 2026 |
| Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. | Andon Labs | Running a business | Everyday | In use | | 6 | 8 | Nov 2025 | Nov 2026 |
| Vending-Bench v1 Replaced in Nov 2025 | Dropped | Grok 4 $4,694 | 1 | Feb 2025 | Nov 2025 | ||||
| BenchCAD 2.1-beta Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. | BenchCAD team | CAD models | Expert | Not in cards | Oct 2026 | ||||
| BenchCAD 2.1 Replaced in Oct 2026 | In use | Claude Fable 5.2 83.5% | 2 | 1 | Oct 2026 | Nov 2026 | |||
| BenchCAD 2.0 Replaced in Oct 2026 | Not in cards | GPT-6 Astra 59% | Oct 2026 | Oct 2026 | |||||
| BenchCAD 1.0 Replaced in Oct 2026 | University | In use | GPT-6 Astra 95.9% | 5 | 6 | May 2026 | Oct 2026 | ||
| ChartQA Pro Model answers 1,948 diverse questions over 1,341 charts, infographics and dashboards, including conversational and unanswerable ones. Accuracy. | vis-nlp (Masry et al.) | Documents, charts and text in images | Everyday | Fading | Claude Opus 4.8 72.3% | 7 | 7 | Apr 2025 | Sep 2026 |
| HiddenMath Models solve 179 competition-level math problems that experts wrote from scratch. Answers are graded automatically. | Google DeepMind | Competition problems with answers | Competition | Dropped | | 7 | 7 | May 2024 | Jul 2026 |
| WenetSpeech Models transcribe Mandarin speech from many domains. Scored by character error rate on the test-net and test-meeting sets. | Northwestern Polytechnical University (ASLP@NPU), Mobvoi and others | Speech recognition and translation | Everyday | Fading | 7 | 7 | Oct 2021 | Sep 2026 | |
| KMMLU-Pro 2026 Models answer questions from the 2026 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. | HAERAE-HUB | Knowledge in many languages | University | Not in cards | o1 78.1% | Oct 2026 | Jul 2025 | ||
| KMMLU-Pro 2025 Replaced in Oct 2026 | Not in cards | Oct 2026 | |||||||
| KMMLU-Pro 2024 exams Replaced in Oct 2026 | LG AI Research and collaborators | Expert-written questions | Expert | In use | | 7 | 6 | Jul 2025 | Aug 2026 |
| CRUXEval Models predict the output of 800 short Python functions for a given input, or an input that yields a given output. Scored by pass@1. | Meta AI and MIT CSAIL | Function-level problems | School | Fading | | 7 | 5 | Jan 2024 | Apr 2026 |
| TempCompass Video models answer questions about temporal aspects of short clips, such as action order, speed, direction and attribute change, in several question formats. | Peking University and Huawei Noah Ark Lab (Liu et al.) | Video understanding | Everyday | Fading | | 7 | 5 | Mar 2024 | Mar 2026 |
| Daily-Omni Models answer 1,197 multiple choice questions about everyday videos that need audio and visual events aligned in time. Scored by accuracy. | Ziwei Zhou, Rui Wang et al. | Video understanding | Everyday | Fading | | 7 | 5 | May 2025 | Sep 2026 |
| PopQA Models answer about 14k entity questions built from Wikidata, spread from popular to rare entities. Scored by whether the answer matches a gold alias. | University of Washington (Mallen et al.) | Trivia and open questions | Everyday | Fading | | 7 | 5 | Dec 2022 | Dec 2025 |
| TVBench Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. | University of Santiago de Compostela and University of Amsterdam et al. | Video understanding | Everyday | In use | | 7 | 4 | Oct 2024 | Aug 2026 |
| FinSearchComp Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. | ByteDance Seed | Finance | Expert | In use | | 7 | 3 | Sep 2025 | Nov 2026 |
| SUNRGBD Models detect objects in about 10,000 indoor RGB-D images and predict their 3D bounding boxes. Scored by average precision. | Princeton University | Detection and captioning | Everyday | Fading | | 7 | 3 | Jun 2015 | Mar 2026 |
| LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. | UT Austin | Simulated robots | Everyday | Fading | | 7 | 3 | Jun 2023 | May 2026 |
| Seal-0 Models answer 111 fact-seeking questions where web search returns conflicting or unhelpful results. Answers are graded for accuracy, usually with search enabled. | Virginia Tech | Questions that need search | University | Fading | Kimi K2.5 57.4% | 7 | 2 | Jun 2025 | Feb 2026 |
| Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Z.ai (zai-org) | Web and app front ends | Expert | In use | Claude Fable 5 82.3% | 7 | 1 | Mar 2026 | Nov 2026 |
| CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). | Alibaba Qwen | Tasks from real jobs | Expert | In use | Claude Fable 5 75.9% | 7 | 1 | Sep 2026 | |
| Big Bench Audio Models answer 1,000 spoken questions from four Big Bench Hard tasks: formal fallacies, navigate, object counting and web of lies. Built to test reasoning over audio input. | Artificial Analysis | Audio understanding and voice | School | In use | Grok Voice 3.0 98.5% | 7 | 1 | Dec 2024 | Oct 2026 |
| Artificial Analysis Coding Agent Index v2 2.0 Coding agents, each a model plus harness, attempt software engineering, terminal, and repository question tasks. The index averages three equal-weight benchmark scores. | Artificial Analysis | Fixing real repositories | In use | | 3 | 1 | Oct 2026 | Nov 2026 | |
| Artificial Analysis Coding Agent Index v1.5 Replaced in Oct 2026 | Not in cards | Sep 2026 | |||||||
| Artificial Analysis Coding Agent Index v1.4 Replaced in Sep 2026 | In use | GPT-6 Astra 67.0 | 1 | Aug 2026 | Sep 2026 | ||||
| Artificial Analysis Coding Agent Index v1.1 Replaced in Aug 2026 | In use | GPT-5.6 Sol 80.0 | 3 | Jun 2026 | Jul 2026 | ||||
| Artificial Analysis Coding Agent Index v1.0 Replaced in Jun 2026 | Not in cards | May 2026 | |||||||
| AlignBench Models answer 683 real Chinese user queries across eight categories. A GPT-4 judge rates each answer from 1 to 10 against a reference. | Tsinghua University and Zhipu AI (THUDM) | Overall chat quality | Everyday | Dropped | | 6 | 13 | Nov 2023 | Sep 2025 |
| CVBench Models answer 2,638 manually checked 2D and 3D vision questions built from ADE20K, COCO, and Omni3D. Scoring is accuracy. | New York University (Cambrian-1 team) | Spatial reasoning and counting | Everyday | Fading | | 6 | 13 | Jun 2024 | Oct 2026 |
| OfficeQA Pro V2 Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match. | Databricks | Documents and spreadsheets | University | In use | Claude Opus 5 54.4% | 1 | Aug 2026 | Oct 2026 | |
| OfficeQA Pro v1 Replaced in Aug 2026 | In use | Claude Fable 5 60.9% | 5 | 11 | Mar 2026 | Nov 2026 | |||
| OfficeQA Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. | Databricks | Documents and spreadsheets | University | In use | Claude Opus 4.7 86.3% | 6 | 11 | Mar 2026 | Nov 2026 |
| FLORES+ plus Models translate English Wikipedia sentences that professionals and community contributors translated into over 200 languages. Outputs are scored with metrics such as chrF against the references. | Open Language Data Initiative (OLDI) | Translation | University | System cards only | | 1 | Oct 2023 | Oct 2026 | |
| FLORES-200 Replaced in Oct 2023 | Meta AI (NLLB Team) | Everyday | Dropped | | 1 | 3 | Jul 2022 | Sep 2025 | |
| Flores v101 Replaced in Jul 2022 | Meta AI | Dropped | | 5 | 5 | Jun 2021 | May 2025 | ||
| FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. | Questions that need search | Everyday | Fading | Grok 4.1 Fast 87.6% | 6 | 6 | Sep 2024 | Feb 2026 | |
| FullStackBench Models write code for 3,374 problems in 16 programming languages across domains like data analysis and web development. Unit tests score each answer. | ByteDance Seed | Function-level problems | University | Fading | | 6 | 5 | Nov 2024 | Feb 2026 |
| WorldSense (omnimodal) Models answer multiple choice questions about real-world videos that need both the visual and the audio track. Scored by accuracy. | Jack Hong, Shilin Yan et al. | Video understanding | Everyday | Fading | | 6 | 5 | Feb 2025 | Sep 2026 |
| CNMO 2024 Models answer problems from the 2024 Chinese National High School Mathematics Olympiad. DeepSeek reports Pass@1 averaged over 16 samples. | Chinese Mathematical Society (contest), DeepSeek (eval set) | Competition problems with answers | Competition | Dropped | | 6 | 5 | Dec 2024 | Jan 2026 |
| Tulu 3 Safety Eval Models answer harmful, jailbreak and benign prompts from several safety benchmarks in Ai2's safety-eval suite. A WildGuard classifier judges the outputs, and the score is an average. | Ai2 | Jailbreaks and prompt injection | Everyday | Fading | | 6 | 5 | Nov 2024 | Apr 2026 |
| PerceptionBench (Moonshot) Moonshot in-house benchmark of atomic visual perception skills. Scoring details unpublished. | Moonshot AI | General visual questions | Everyday | In use | GPT-5.6 Sol 59.7% | 6 | 3 | Jul 2026 | Nov 2026 |
| VQAv2 Open-ended questions about COCO images, balanced to reduce language priors. Scored by consensus accuracy against 10 human answers. | Virginia Tech + Georgia Tech | General visual questions | Everyday | Fading | Reka Edge 88.4% | 6 | 3 | Dec 2016 | Mar 2026 |
| ResearchRubrics 101 deep-research prompts with 2,593 expert-written weighted rubric criteria (including penalties). Reports graded for grounding, reasoning and clarity. | Scale AI | Deep research | Expert | In use | | 6 | 3 | Nov 2025 | Nov 2026 |
| MuSR Models answer questions about long generated narratives such as murder mysteries, object placement and team allocation. Each item needs several soft reasoning steps over the story. | UT Austin | Language reasoning | University | Fading | | 6 | 2 | Oct 2023 | Dec 2025 |
| CL-bench Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. | Tencent Hunyuan (with Fudan University authors) | Learning from the context | University | In use | GPT-5.4 27.9% | 6 | 2 | Feb 2026 | Jul 2026 |
| TIR-Bench Models solve 13 kinds of image tasks that need tool use on images, such as rotation, jigsaw, and maze. Most items are multiple choice. | Agents-X project (Ming Li et al.) | General visual questions | Everyday | Fading | | 6 | Nov 2025 | Apr 2026 | |
| QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. | Qwen Team, Alibaba Group | General assistants | Everyday | Fading | Claude Sonnet 4.6 60.5% | 6 | Apr 2026 | Jun 2026 | |
| Full-Duplex-Bench v3 3.0 Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking. | Guan-Ting Lin, Hung-yi Lee and collaborators (National Taiwan University and others) | Speech recognition and translation | Everyday | Fading | | 1 | Apr 2026 | May 2026 | |
| Full-Duplex-Bench v2 2.0 Replaced in Apr 2026 | Not in cards | gpt-realtime 4.05 | Oct 2025 | Aug 2025 | |||||
| Full-Duplex-Bench v1.5 Replaced in Oct 2025 | Fading | | 2 | Jul 2025 | Jul 2026 | ||||
| Full-Duplex-Bench 1.0 Replaced in Jul 2025 | In use | | 3 | Mar 2025 | Oct 2026 | ||||
| CommonsenseQA 2.0 Models answer 14,343 yes or no commonsense questions that players wrote to fool an AI in a game. Scored by accuracy. | Allen Institute for AI and Tel Aviv University | Commonsense | Everyday | Not in cards | | Jan 2022 | Jul 2023 | ||
| CommonSenseQA 1.0 Replaced in Jan 2022 | Tel Aviv University and Allen Institute for AI | Dropped | PaLM 2-L 90.4% | 5 | 21 | Nov 2018 | Oct 2026 | ||
| PaperBench Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. | OpenAI | AI research tasks | Research frontier | In use | | 5 | 12 | Apr 2025 | Nov 2026 |
| CLUEWSC Models decide whether a pronoun in a Chinese sentence refers to a given noun phrase, a Winograd-style task. Scored by accuracy. | CLUE benchmark team | Reasoning in many languages | Everyday | Fading | | 5 | 7 | Mar 2020 | Apr 2026 |
| FORTRESS Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. | Scale AI | Refusing harmful requests | In use | Claude Opus 5.5 5.78% | 5 | 6 | Jun 2025 | Nov 2026 | |
| MME Models answer yes or no questions about images across 14 perception and cognition subtasks with hand-written instructions. Labs usually report the perception sum, out of 2000. | Fu et al. (Tencent Youtu Lab and Xiamen University) | General visual questions | Everyday | Fading | | 5 | 6 | Jun 2023 | Aug 2026 |
| VisuLogic Models answer 1,000 visual reasoning puzzles in six categories that cannot be solved by describing the image in words. Scored by accuracy. | Shanghai AI Laboratory with several universities | Spatial reasoning and counting | Everyday | Fading | | 5 | 6 | Apr 2025 | Jun 2026 |
| Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. | Vals AI | Composite work indices | In use | Claude Fable 5.2 71.3% | 4 | 3 | Oct 2026 | Nov 2026 | |
| Vals Index v2.1 Replaced in Oct 2026 | In use | | 1 | 1 | Sep 2026 | Oct 2026 | |||
| Vals Index v2 Replaced in Sep 2026 | Not in cards | Claude Opus 5.5 69.7% | Aug 2026 | Sep 2026 | |||||
| Vals Index v1 Replaced in Aug 2026 | System cards only | Claude Opus 4.7 72.2% | 1 | Oct 2025 | Jul 2026 | ||||
| USAMO 2026 Models write full proofs for the six problems of the 2026 USA Mathematical Olympiad. Judges grade each proof from 0 to 7 against rubrics. | Mathematical Association of America (contest), MathArena (evaluation) | Written proofs | Competition | In use | GPT-5.5 98.2% | 3 | 5 | Mar 2026 | Jul 2026 |
| USAMO 2025 Replaced in Mar 2026 | MathArena (ETH Zurich SRI Lab) | Dropped | | 2 | Mar 2025 | Jul 2025 | |||
| USAMO 2024 Replaced in Mar 2025 | MathArena (ETH Zurich SRI Lab) | Not in cards | Mar 2024 | ||||||
| PutnamBench Models write formal Lean proofs of Putnam competition problems. A proof counts only when the verifier accepts it. | UT Austin (Tsoukalas et al.) | Formal proofs | Competition | Fading | | 5 | 5 | Jul 2024 | Jul 2026 |
| Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness. | Google DeepMind | Function-level problems | School | Dropped | | 5 | 5 | Dec 2023 | Jul 2026 |
| Design2Code Models turn screenshots of 484 real webpages into HTML and CSS. Automatic metrics compare the rendered page with the original on layout, text, position, color and CLIP similarity. | Stanford SALT and others (Si, Zhang, Yang et al.) | Web and app front ends | Everyday | Fading | GPT-4V 87.2 | 5 | 5 | Mar 2024 | Apr 2026 |
| Seed-TTS-eval Models clone a voice from a short prompt and speak new text in English and Mandarin. Scored by word error rate from an ASR model and by speaker similarity. | ByteDance (Seed-TTS team) | Speech recognition and translation | Everyday | Fading | | 5 | 5 | Jun 2024 | Sep 2026 |
| ArtifactsBench (Gemini 2.5 Pro 0506 judge) Models generate interactive visual code artifacts for 1,825 tasks. An MLLM judge scores rendered screenshots and code against a per-task checklist. | Tencent Hunyuan | Web and app front ends | University | Not in cards | | Jul 2025 | Jul 2025 | ||
| ArtifactsBench Gemini 2.5 Pro judge Replaced in Jul 2025 | In use | | 5 | 4 | Jul 2025 | Jul 2026 | |||
| Audio MC Spoken multi-turn conversations test inference memory, instruction retention, self-coherence and voice editing. O4-mini judge applies per-conversation rubrics. | Scale AI | Audio understanding and voice | Everyday | In use | | 5 | 4 | Dec 2025 | Oct 2026 |
| miniF2F Models write formal proofs of olympiad and high school competition statements, such as AMC, AIME and IMO problems. A proof assistant checks each proof, and scores are pass@k on the 244 test problems. | OpenAI (Kunhao Zheng, Jesse Michael Han, Stanislas Polu) | Formal proofs | Competition | Fading | | 5 | 4 | Aug 2021 | Jul 2026 |
| FActScore Models write biographies of people, and each output is split into atomic facts. The score is the share of facts that a knowledge source supports, as the paper defines FActScore. | Sewon Min et al. (University of Washington and collaborators) | Long-form factuality | Everyday | Fading | | 5 | 4 | May 2023 | Feb 2026 |
| AdvancedIF Models respond to over 1,600 prompts with complex, multi-turn, and system-level instructions. A judge model checks each response against expert-written rubrics. | Meta Superintelligence Labs | Following precise instructions | Everyday | In use | | 5 | 3 | Nov 2025 | Sep 2026 |
| GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. | Meta AI, Hugging Face, and AutoGPT | General assistants | Everyday | Fading | GPT-5 66.1% | 5 | 3 | Nov 2023 | Mar 2026 |
| CMATH 1.7K Chinese elementary-school math word problems. Final-answer accuracy. | Xiaomi AI Lab | Word problems | School | Fading | | 5 | 3 | Jun 2023 | Apr 2026 |
| MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. | ByteDance Seed and collaborators (Li et al.) | Deep research | Expert | Fading | GPT-5 24.3% | 5 | 2 | Aug 2025 | Jun 2026 |
| SpreadsheetBench Real spreadsheet manipulation questions from Excel forums. Output workbook is checked cell-by-cell against reference solutions. | Renmin University (RUCKBReasoning) | Documents and spreadsheets | University | In use | | 5 | 2 | Jun 2024 | Jun 2026 |
| WebArena-Verified Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. | ServiceNow | Web browsing | Everyday | In use | | 5 | 2 | Dec 2025 | Oct 2026 |
| MLS-Bench Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. | MLS-Bench authors (Bohan Lyu et al.) | AI research tasks | Research frontier | In use | Claude Fable 5.1 50.3% | 5 | 1 | May 2026 | Sep 2026 |
| ToolSandbox Models use stateful tools in multi-turn conversations with a simulated user. Tools have hidden state dependencies, and runs are scored against milestones. | Apple | General assistants | Everyday | In use | LFM2.5-24B-A2B 80.7% | 5 | 1 | Aug 2024 | Oct 2026 |
| LongFact Models write long answers to 2,280 fact-seeking prompts on 38 topics. A search-backed checker verifies each claim, and cards report the hallucination rate. | Google DeepMind | Long-form factuality | Fading | GPT-5 mini 0.70% | 5 | 1 | Mar 2024 | Jun 2026 | |
| AttaQ Models answer 1,402 adversarial questions in seven harm classes, such as deception, violence and PII. Responses are scored for harmlessness. | IBM Research | Refusing harmful requests | Everyday | Fading | Granite 3.3 8B Instruct 88.5% | 5 | Sep 2023 | Apr 2026 | |
| Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. | OpenAI | Finance | Expert | Fading | GPT-5.5 Pro 88.6% | 5 | Jul 2025 | Apr 2026 | |
| DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring. | DeepSeek | Building whole projects | Expert | In use | | 5 | Aug 2026 | Aug 2026 | |
| StrongREJECT Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. | UC Berkeley (Souly et al.) | Jailbreaks and prompt injection | In use | | 4 | 31 | Feb 2024 | Nov 2026 | |
| Needle In A Haystack Models retrieve a planted fact from long filler text at varied context lengths and insertion depths. Scored by retrieval accuracy. | Greg Kamradt | Finding facts in long inputs | Everyday | Dropped | GLM-4 100% | 4 | 11 | Nov 2023 | Jan 2025 |
| DRACO 100 deep-research tasks from real Perplexity queries across 10 domains, graded on about 40 expert rubric criteria each. | Perplexity | Deep research | Expert | In use | Claude Opus 5 88.6% | 4 | 8 | Feb 2026 | Nov 2026 |
| AISHELL-2 Models transcribe 1,000 hours of Mandarin speech recorded on several devices. Scored by character error rate. | AISHELL (Beijing Shell Shell Technology) | Speech recognition and translation | Everyday | Dropped | Kimi-Audio-7B-Instruct 2.56% | 2 | 3 | Aug 2018 | Feb 2026 |
| AISHELL-1 Replaced in Aug 2018 | Dropped | | 2 | 4 | Sep 2017 | Feb 2026 | |||
| MM-MT-Bench Models hold multi-turn conversations about images such as charts, documents and photos. GPT-4o grades each answer against a reference on a 1 to 10 scale. | Mistral AI | General visual questions | Everyday | Dropped | | 4 | 5 | Oct 2024 | Jan 2026 |
| SysBench Models follow 500 system messages across five user turns each. A model verifier checks constraints, instructions and whole sessions. | Peking University + Baichuan | Following precise instructions | Everyday | Fading | GLM-Z1-32B-0414 81.2% | 4 | 5 | Aug 2024 | Oct 2026 |
| WebArena Agents complete 812 realistic tasks on self-hosted shopping, forum, GitLab, CMS and map websites. Functional graders check the outcome. | Carnegie Mellon University | Web browsing | Everyday | Fading | Claude Opus 4.6 68% | 4 | 5 | Jul 2023 | Oct 2026 |
| OSWorld-G Models map 564 desktop instructions to the right screen element or point, covering text matching, element recognition, layout understanding and fine manipulation. | XLANG Lab, University of Hong Kong (OSWorld team) | Finding things on screen | Everyday | Fading | | 4 | 5 | May 2025 | Jun 2026 |
| RACE-H Models answer multiple-choice reading questions from English exams for Chinese high school students. The test split has 3,498 questions, scored by accuracy. | Carnegie Mellon University | Reading comprehension | School | Dropped | Claude 3 Opus 92.9% | 4 | 5 | Apr 2017 | Dec 2024 |
| CWE-bench 2 Agents audit and patch vulnerabilities in 250 codebases. The score is pass@1 under deterministic verification. | Collinear AI | Secure code and review | Expert | In use | Claude Fable 5.2 75% | 1 | 1 | Nov 2026 | Nov 2026 |
| CWE-bench v1 Replaced in Nov 2026 | In use | | 3 | 3 | Sep 2026 | Nov 2026 | |||
| TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. | Yale University and Allen Institute for AI | Video understanding | Everyday | Fading | | 4 | 4 | Oct 2024 | Jun 2026 |
| AMO-Bench Models solve 50 original math problems at or above IMO difficulty that need only a final answer. Answers are graded automatically for accuracy. | Meituan LongCat Team | Competition problems with answers | Competition | Fading | GPT-5 52.4% | 4 | 4 | Oct 2025 | Jun 2026 |
| Riemann-bench 25 private research-mathematics problems from professors and IMO medalists. Agents with tools, 100 runs per problem. | Surge AI | Research mathematics | Research frontier | In use | GPT-5.6 Sol 83.2% | 4 | 4 | Apr 2026 | Nov 2026 |
| OVBench Video models answer questions about past, current and future moments of an online video stream in 6 task types and 16 subtasks. | Nanjing University MCG (Huang et al.) | Video understanding | Everyday | Fading | | 4 | 4 | Dec 2024 | Jun 2026 |
| Video-Holmes Models answer 1,837 questions about 270 suspense short films across seven reasoning tasks. Each question needs several visual clues from different parts of the film. | Tencent ARC Lab and City University of Hong Kong | Video understanding | Everyday | Fading | | 4 | 4 | May 2025 | Jun 2026 |
| MMSU Models answer questions about spoken language that need its content, its paralinguistic cues such as emotion and pitch, and its phonology. | Wang et al. (arXiv 2506.04779) | Speech recognition and translation | University | Fading | | 4 | 4 | Jun 2025 | Sep 2026 |
| OVO-Bench Video models answer questions asked at given timestamps of a video stream in 12 tasks over 644 videos. Tasks cover backward tracing, real-time understanding and forward active responding. | Niu et al. (CVPR 2025) | Video understanding | Everyday | Fading | | 4 | 4 | Jan 2025 | Jun 2026 |
| WMT26 v2026 Systems translate the WMT26 general task test set in 24 directions, mostly out of English, across several domains with added instructions. The shared task ranks systems by human contrastive ESA judgments. | WMT (Conference on Machine Translation) | Translation | University | In use | | 1 | 1 | Jun 2026 | Sep 2026 |
| WMT25 v2025 Replaced in Jun 2026 | System cards only | | 3 | Jun 2025 | Sep 2026 | ||||
| WMT24 v2024 Replaced in Jun 2025 | Not in cards | Jun 2024 | |||||||
| WMT23 v2023 Replaced in Jun 2024 | Dropped | | 3 | Jul 2023 | May 2024 | ||||
| Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. | Shanghai AI Lab (Yuhao Zhou et al.) | General visual questions | Expert | Fading | | 4 | 4 | Jun 2025 | Sep 2026 |
| CLIcK Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. | KAIST and collaborators (Eunsu Kim et al.) | Knowledge in many languages | School | In use | | 4 | 4 | Mar 2024 | Aug 2026 |
| WildClawBench 60 original in-the-wild tasks run inside a live OpenClaw assistant environment, chaining 10-60+ tool calls. Scored by task-level graders. | InternLM (Shanghai AI Lab) | General assistants | University | In use | GPT-5.6 Sol 67.2% | 4 | 3 | Mar 2026 | Oct 2026 |
| VLMsAreBiased Models count or identify features in altered images of famous subjects, such as a logo with an added stripe, across 7 domains. Scoring is accuracy. | An Vo et al. | Spatial reasoning and counting | Everyday | Fading | GPT-6 Astra 84.1% | 4 | 3 | May 2025 | Sep 2026 |
| VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output. | MiniMax | Building whole projects | Expert | In use | | 1 | Jun 2026 | Jun 2026 | |
| VIBE-Pro (MiniMax) Replaced in Jun 2026 | Fading | | 2 | 3 | Feb 2026 | May 2026 | |||
| VIBE (MiniMax) v1 Replaced in Feb 2026 | Fading | Claude Opus 4.5 90.7% | 1 | Dec 2025 | Dec 2025 | ||||
| Flame-React-Eval Models write React components from UI design images. Rendered outputs are compared with gold components by image similarity and scored as pass@k. | Flame-Code-VLM project (Ge et al.) | Web and app front ends | University | Fading | GLM-5V-Turbo 93.8% | 4 | 3 | Feb 2025 | Apr 2026 |
| MMAR Models answer 1,000 questions about real-world audio of speech, sound, music and their mixes. Questions span four reasoning layers from signal to cultural knowledge. | Ma et al. (arXiv 2505.13032) | Audio understanding and voice | University | In use | | 4 | 3 | May 2025 | Oct 2026 |
| Harvey LAB-AA v1.1 Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed. | Artificial Analysis | Law | Expert | In use | GPT-6.2 Astra 15% | 4 | 3 | Oct 2026 | Nov 2026 |
| HumanEval Multi Models complete HumanEval problems in Python and seven more languages, including C++, Java, PHP, and Bash. Unit tests score pass@1. | DeepSeek-AI | Function-level problems | School | Dropped | | 4 | 3 | Nov 2023 | Jan 2025 |
| OSWorld 2.1 (offline subset) Agents complete the 82 OSWorld 2.1 tasks on the official offline list, with no internet access. Scored by checkpoint partial credit and strict pass rate. | XLANG Lab, HKU | Desktop apps | University | In use | Claude Fable 5.2 89.3% | 4 | 3 | Sep 2026 | Nov 2026 |
| MMLongBench Vision-language models solve 13,331 long-context examples in five task types, such as visual RAG, needle search and many-shot in-context learning. Each example comes at five lengths from 8K to 128K tokens. | University of Edinburgh and collaborators (Wang et al.) | Finding facts in long inputs | Everyday | Fading | | 4 | 3 | May 2025 | Jun 2026 |
| MuChoMusic Models answer multiple-choice questions about music recordings. It tests music knowledge and reasoning in audio-language models. | Weck et al. (Universitat Pompeu Fabra, Queen Mary University of London and others) | Audio understanding and voice | University | Fading | | 4 | 3 | Aug 2024 | Sep 2026 |
| EQ-Bench 4 Models hold 16-message roleplay chats with 120 simulated user personas whose preferences compete and are hidden. A three-judge panel compares transcripts pairwise, and ratings are fitted as Elo scores. | EQ-Bench (Sam Paech) | Multi-turn conversation | Everyday | Not in cards | Claude Opus 5 1,365 | Jul 2026 | Oct 2026 | ||
| EQ-Bench 3 Replaced in Jul 2026 | EQ-Bench | Fading | Claude Fable 5 2,096 | 4 | 2 | Apr 2025 | Oct 2026 | ||
| EQ-Bench v2 Replaced in Apr 2025 | Overall chat quality | Not in cards | Claude 3.5 Sonnet 86.4 | Jan 2024 | Mar 2025 | ||||
| MathArena Apex Shortlist Models solve 47 hard final-answer problems that MathArena selected from 2025 competitions, where the best models scored about 50% at selection. Scored by accuracy over repeated attempts. | ETH Zurich SRI Lab | Competition problems with answers | Competition | In use | GPT-5.5 98.4% | 4 | 2 | Aug 2025 | Oct 2026 |
| KOR-Bench Models read a new rule set and apply it to questions in five task types: operation, logic, cipher, puzzle and counterfactual. Scored by accuracy against an answer key. | Multimodal Art Projection (M-A-P) with ByteDance and 01.AI | Abstract puzzles and logic | University | Fading | o1-preview 72.9% | 4 | 2 | Oct 2024 | Mar 2026 |
| KorMedMCQA Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. | KAIST and collaborators (Sunjun Kweon et al.) | Medicine | Expert | In use | o1-preview 92.7% | 4 | 2 | Mar 2024 | Aug 2026 |
| SUPERChem Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. | Zhao et al. | Genomics and life sciences | Expert | In use | | 4 | 2 | Dec 2025 | Aug 2026 |
| OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. | Renmin University of China (RUC-NLPIR) and others | Deep research | University | Fading | | 4 | 2 | Feb 2026 | Sep 2026 |
| QIVD Models answer spoken questions about short phone videos in real time, as in a face to face talk. Each item has a short answer and a timestamp. | Qualcomm AI Research and University of Toronto | Video understanding | Everyday | Fading | | 4 | 2 | Mar 2025 | Sep 2026 |
| xbench-DeepSearch-2510 Agents answer 100 new Chinese deep search questions that need many web searches and cross-checks. Answers are matched to a key. | xbench (HongShan) | Deep research | Expert | Fading | | 1 | 1 | Oct 2025 | Feb 2026 |
| xbench-DeepSearch v2505 Replaced in Oct 2025 | Fading | | 3 | 1 | May 2025 | Feb 2026 | |||
| FLTEval Agents complete the formal Lean proofs and define the new mathematical concepts in real pull requests to the Fermat's Last Theorem (FLT) formalization project. Lean and SafeVerify check each submitted diff. | Mistral AI | Formal proofs | Research frontier | In use | | 4 | 2 | Mar 2026 | Jul 2026 |
| WildSpeech-Bench Models hold spoken conversations built from real chat data, with prosody, homophones and stuttering. A judge model scores the replies. | Zhang et al. (arXiv 2506.21875) | Speech recognition and translation | Everyday | Fading | | 4 | 2 | Jun 2025 | Sep 2026 |
| KBL Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items. | LBox and collaborators (Yeeun Kim et al.) | Law | Expert | In use | | 4 | 2 | Oct 2024 | Aug 2026 |
| MTEB Massive Text Embedding Benchmark: retrieval, clustering, classification across languages and modalities. | MTEB community | Other | Fading | | 4 | 2 | Oct 2022 | Oct 2026 | |
| WildBench 1,024 challenging real-user tasks with checklist-guided LLM judging. | Ai2 | Overall chat quality | University | Dropped | | 4 | 1 | Jun 2024 | Jan 2026 |
| FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. | Google DeepMind + Google Research + Kaggle | Short factual questions | University | Fading | | 4 | 1 | Dec 2025 | Oct 2026 |
| SWE-Bench Pro V2 Hard Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol. | Scale AI (co-developed with Reflection) | Fixing real repositories | Research frontier | In use | Claude Opus 5 98% | 4 | 1 | Sep 2026 | Nov 2026 |
| EnterpriseOps-Gym 1.1 Agents complete 296 workflows across customer service, human resources, and IT service management. SQL checks grade the resulting database state. | ServiceNow AI Research | Apps and MCP servers | Expert | Not in cards | Nov 2026 | ||||
| EnterpriseOps-Gym Original Replaced in Nov 2026 | ServiceNow | In use | Claude Opus 4.6 45.9% | 4 | 1 | Mar 2026 | Nov 2026 | ||
| GeneBench Pro Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. | OpenAI | Genomics and life sciences | Expert | In use | GPT-5.6 Sol 31.5% | 4 | 1 | Jun 2026 | Oct 2026 |
| MedChemBench (OpenAI) Models work through 99 medicinal chemistry problems drawn from recent papers and patents. Each task mixes prose with structures, assay tables and plots, then asks for the program’s next decision. | OpenAI | Genomics and life sciences | Expert | In use | GPT-6 Astra 83.8% | 4 | 1 | Jul 2026 | Oct 2026 |
| LifeSciBench Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. | OpenAI | Genomics and life sciences | Expert | In use | GPT-Rosalind-6 61.3% | 4 | 1 | Jun 2026 | Oct 2026 |
| RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. | NVIDIA + UT Austin (RoboCasa team) | Simulated robots | Everyday | Fading | | 4 | 1 | Mar 2025 | Jul 2026 |
| Internal API instruction following (OpenAI) Models answer prompts drawn from real developer use cases that carry instructions on formatting, verbosity and length. OpenAI reports a hard subset. | OpenAI | Following precise instructions | Everyday | Dropped | GPT-4.5 54% | 4 | Apr 2025 | Aug 2025 | |
| τ-Voice Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction. | Sierra | Speech recognition and translation | Everyday | In use | GPT-Live-1 81.7% | 4 | Mar 2026 | Oct 2026 | |
| RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. | UT Austin + NVIDIA (RoboCasa team) | Simulated robots | Everyday | Not in cards | | Mar 2026 | Dec 2025 | ||
| RoboCasa Original Replaced in Mar 2026 | Fading | | 4 | Jun 2024 | Jul 2026 | ||||
| QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings. | Qwen | Web and app front ends | University | Fading | | 4 | Apr 2026 | Apr 2026 | |
| MLE-bench Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. | OpenAI | Machine learning engineering | Expert | In use | GPT-6.2 Astra 96.1% | 3 | 18 | Oct 2024 | Nov 2026 |
| SQuAD v2 2.0 Models extract answers from Wikipedia paragraphs and must abstain on over 50,000 adversarial unanswerable questions. Scored by exact match and F1. | Stanford NLP | Reading comprehension | Everyday | Dropped | | 2 | 7 | Jun 2018 | Nov 2024 |
| SQuAD 1.1 Replaced in Jun 2018 | Dropped | | 1 | 5 | Jun 2016 | Jul 2024 | |||
| Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. | Irregular | Capture-the-flag challenges | Expert | In use | GPT-6 Astra 90.9% | 3 | 11 | Dec 2025 | Nov 2026 |
| Gray Swan IPI Arena benchmark Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. | Gray Swan + UK AISI + US CAISI | Jailbreaks and prompt injection | In use | | 3 | 10 | Mar 2026 | Nov 2026 | |
| LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. | Edison Scientific (FutureHouse) | Lab and data work | Expert | In use | GPT-5.2 60.9% | 3 | 4 | Feb 2026 | Oct 2026 |
| LAB-Bench v1 Replaced in Feb 2026 | FutureHouse | Genomics and life sciences | System cards only | Grok 4.6 80.7% | 4 | Jul 2024 | Sep 2026 | ||
| ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. | OpenAI | Finding and exploiting real bugs | Research frontier | In use | GPT-6.2 Astra 53.5% | 3 | 8 | Sep 2026 | Nov 2026 |
| POPE Vision language models answer yes or no questions about whether objects appear in COCO images. Scored by accuracy or F1 on object hallucination. | Renmin University of China (Li et al.) | Hallucination rates | Everyday | Fading | | 3 | 6 | May 2023 | Aug 2026 |
| All-Angles Bench Models answer over 2,100 multi-view questions about 90 real scenes. Six tasks cover counting, attributes, relative distance and direction, object manipulation and camera pose. | Yeh et al. (UC Berkeley, NYU and others) | Spatial reasoning and counting | Everyday | Fading | | 3 | 6 | Apr 2025 | Feb 2026 |
| ArXivMath Final-answer research-level math problems extracted monthly from recent arXiv papers and filtered to be self-contained. | MathArena (ETH Zurich SRI Lab) | Research mathematics | Research frontier | In use | | 3 | 6 | Feb 2026 | Oct 2026 |
| OmniSpatial Vision language models answer over 8,400 spatial reasoning questions in four groups: dynamic reasoning, complex spatial logic, spatial interaction and perspective taking. | Jia, Qi et al. (OmniSpatial authors) | Spatial reasoning and counting | Everyday | Fading | | 3 | 6 | Jun 2025 | Oct 2026 |
| MMBench-CN v1.1 Models answer the Chinese translation of the MMBench image questions, using CircularEval with shuffled options. Scored by accuracy. | Shanghai AI Laboratory (OpenCompass) | Knowledge in many languages | Everyday | Fading | | 3 | 5 | Mar 2026 | |
| GPQA Main Models answer the 448 question main set of GPQA, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. | NYU (Rein et al.) | Science questions | Expert | Dropped | | 3 | 5 | Nov 2023 | Nov 2025 |
| MMSI-Bench Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data. | Shanghai AI Lab (OpenRobotLab) and collaborators | Spatial reasoning and counting | Everyday | Fading | | 3 | 5 | May 2025 | Oct 2026 |
| ChartMuseum Models answer 1,162 expert-annotated questions about real-world charts from 184 sources. Many questions need visual reasoning that text cannot replace. | UT Austin (Tang et al.) | Documents, charts and text in images | Expert | Fading | | 3 | 5 | May 2025 | Jun 2026 |
| KMMLU Models answer 35,030 Korean multiple-choice questions across 45 subjects, collected from original Korean exams. Scoring is accuracy. | HAE-RAE | Knowledge in many languages | University | Fading | | 3 | 5 | Feb 2024 | Jan 2026 |
| MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. | Google DeepMind | Long videos | University | Fading | | 3 | 5 | May 2025 | Jun 2026 |
| ActivityNet-QA Models answer open-ended questions about web videos of human activities, from 58,000 QA pairs on 5,800 videos. Scored by accuracy. | Hangzhou Dianzi University (MILVLG) | Video understanding | Everyday | Dropped | | 3 | 5 | Jun 2019 | Jul 2025 |
| QVHighlights Natural-language queries over YouTube videos. Model localizes relevant moments, scored by Recall@1 at IoU 0.5. | UNC Chapel Hill | Video understanding | Everyday | Fading | | 3 | 5 | Jul 2021 | Dec 2025 |
| MTOB Models learn to translate between English and Kalamang, a language with under 200 speakers, from one grammar book. Scored by chrF. | Harvard University and collaborators | Learning from the context | Expert | Dropped | | 3 | 5 | Sep 2023 | Jul 2026 |
| PHYBench Models solve 500 original physics problems from high school to Physics Olympiad level. Answers are expressions checked against the key. | Qiu et al. (PHYBench team) | Science olympiads | Competition | Fading | | 3 | 4 | Apr 2025 | Jul 2026 |
| IMO-ProofBench Models write proofs for 60 olympiad problems, split into a basic set and an advanced set up to IMO-hard. Each proof is graded from 0 to 7 by experts or a Gemini autograder. | Google DeepMind (Superhuman Reasoning team) | Written proofs | Competition | Fading | GPT-5.5 Pro 88.1% | 3 | 4 | Nov 2025 | Sep 2026 |
| EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. | Google DeepMind and University of Bristol | Video understanding | Everyday | Fading | | 3 | 4 | Mar 2025 | Apr 2026 |
| HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. | SKYLENAGE AI | Expert-written questions | Expert | In use | | 3 | 4 | Feb 2026 | Sep 2026 |
| Blueprint-Bench 2 Agents turn about 20 interior photos of each of 50 apartments into 2D floor plans. Scores compare room connectivity with ground truth, normalized against random. | Andon Labs | Spatial reasoning and counting | Everyday | In use | | 3 | 3 | May 2026 | Nov 2026 |
| Blueprint-Bench v1 Replaced in May 2026 | Not in cards | GPT-5 0.43 | Sep 2025 | Sep 2025 | |||||
| Video SimpleQA Video models answer multi-hop fact-seeking questions that need knowledge beyond the video and evidence from its time segments. Short definitive answers are graded for factual correctness. | Cao et al. (Video SimpleQA team) | Short factual questions | University | Fading | o3 66.3% | 3 | 3 | Mar 2025 | Jun 2026 |
| Inverse IFEval Models follow 1,012 Chinese and English instructions that conflict with habits learned in fine-tuning, such as writing code without comments. A model judge scores compliance. | ByteDance Seed and collaborators (Zhang et al.) | Following precise instructions | Everyday | Fading | o3 76.1% | 3 | 3 | Sep 2025 | Oct 2026 |
| OCR-Reasoning Models answer 1,069 human-annotated questions that need reasoning over text-rich images. The questions cover 6 reasoning abilities and 18 practical tasks. | SCUT-DLVCLab (Huang et al.) | Documents, charts and text in images | Everyday | Fading | o1 44.4% | 3 | 3 | May 2025 | Oct 2026 |
| AceBench Models call tools from 4,538 Chinese and English APIs in normal, special, and multi-turn agent cases. Answers are checked without LLM judges or live APIs. | Chen Chen et al. | Function calling | Everyday | Fading | GPT-4o 85.4% | 3 | 3 | Jan 2025 | Feb 2026 |
| KoBALT Models answer 700 expert-written multiple-choice questions on 24 Korean linguistic phenomena in syntax, semantics, pragmatics, phonology and morphology. | Seoul National University and collaborators (Hyopil Shin et al.) | Reasoning in many languages | University | Fading | Claude 3.7 Sonnet 61% | 3 | 3 | May 2025 | Apr 2026 |
| Omni-MATH Models solve 4,428 Olympiad-level competition math problems across 33 subdomains and 10 difficulty levels. A GPT-4 or Omni-Judge grader checks answers. | Peking University and Alibaba et al. | Competition problems with answers | Competition | Dropped | o1-mini 60.5% | 3 | 3 | Oct 2024 | Oct 2025 |
| tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. | Amazon AGI | Customer service | Everyday | Fading | Grok 4.1 Fast 82.7% | 3 | 3 | Dec 2025 | Dec 2025 |
| HRM8K Models solve 8,011 math problems given as English and Korean parallel pairs. It measures how much math reasoning drops when problems are in Korean. | HAE-RAE and OneLineAI (Hyunwoo Ko et al.) | Word problems | School | Fading | | 3 | 3 | Jan 2025 | Jul 2026 |
| XLRS-Bench Models answer perception and reasoning questions about extremely large, ultra-high-resolution remote sensing images. | XLRS-Bench team (Fengxiang Wang et al.) | Detection and captioning | Expert | Fading | Qwen2-VL-7B-Instruct 41.1% | 3 | 3 | Mar 2025 | Sep 2026 |
| OlympiadBench Models solve bilingual olympiad-level math and physics problems in Chinese and English, many with images. Answers are checked against a key. | OpenBMB and Tsinghua University (He et al.) | Competition problems with answers | Competition | Dropped | GPT-4V 18% | 3 | 3 | Feb 2024 | Aug 2025 |
| StreamingBench Models answer questions while a video streams in, as a human viewer would. It tests real-time visual understanding, omni-source understanding and context understanding. | Lin et al. (arXiv 2411.03628) | Video understanding | Everyday | Fading | | 3 | 3 | Nov 2024 | Sep 2026 |
| WebTailBench Computer-use agents complete 609 hand-verified tasks on live websites across 11 categories, such as bookings, real estate and job search. A companion set of 111 harmful tasks tests refusals. | Microsoft | Web browsing | Everyday | Fading | GPT-5.4 57.4% | 3 | 3 | Nov 2025 | Jul 2026 |
| Putnam 2025 Models solve the 12 problems of the 2025 William Lowell Putnam competition with full proofs. Scored out of 120 points. | Mathematical Association of America | Written proofs | Competition | Fading | | 2 | 2 | Dec 2025 | Jun 2026 |
| Putnam 2024 Replaced in Dec 2025 | Dropped | | 1 | 1 | Dec 2024 | Nov 2025 | |||
| VATEX Models write English or Chinese captions for short video clips from a set of over 41,000 videos. Scored by CIDEr. | UC Santa Barbara and ByteDance AI Lab | Video captioning | Everyday | Fading | | 3 | 3 | Apr 2019 | May 2026 |
| Windows Agent Arena Agents complete more than 150 tasks in a real Windows OS with its apps, tools and browsers. Scripts adapted from OSWorld check the final state. | Microsoft | Desktop apps | Everyday | Dropped | | 3 | 3 | Sep 2024 | Nov 2025 |
| Charades-STA Models read a sentence describing an action and return its start and end time in a Charades indoor video. Scored by temporal IoU with the labeled moment. | University of Southern California (Gao, Sun, Yang, Nevatia) | Video understanding | Everyday | Fading | | 3 | 3 | May 2017 | May 2026 |
| DUDE Models answer extractive, abstractive, and unanswerable questions about multi-page, multi-domain documents. Scored by ANLS. | DUDE team (KU Leuven, CVC Barcelona, et al.) | Documents, charts and text in images | Everyday | Fading | | 3 | 3 | May 2023 | Jun 2026 |
| GenEval Text-to-image models draw prompts about object count, color, position and co-occurrence, and an object detector checks each image. Scored by the share of correct images. | University of Washington (Ghosh et al.) | Image generation and editing | Everyday | Fading | | 3 | 3 | Oct 2023 | Feb 2026 |
| Ko-LongBench (LG) Models answer questions over long Korean contexts. Tasks cover document QA, story and dialogue history understanding, in-context learning, structured QA and RAG. | LG AI Research | Reasoning over long documents | University | Fading | | 3 | 3 | Jul 2025 | Aug 2026 |
| PhiBench Models answer original prompts that Microsoft's Phi team wrote to test a wide range of skills and reasoning. | Microsoft | Other | University | Dropped | | 3 | 3 | Dec 2024 | Apr 2025 |
| RealKIE-FCC Verified Model extracts structured fields from scanned FCC invoices into a JSON schema. Amazon human-verified labels. Field-level similarity (ALNS). | Indico Data (RealKIE) + Amazon AGI (Verified) | Documents, charts and text in images | Everyday | Fading | | 3 | 3 | Dec 2025 | Dec 2025 |
| WebQuest Models answer questions from screenshots of one web page, several related pages, or a navigation trace. The test checks extraction and combination of information across pages. | Google (Wang et al.) | Documents, charts and text in images | Everyday | Fading | GLM-4.6V 69.3% | 3 | 3 | Sep 2024 | Jan 2026 |
| Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. | Meta (Meta Superintelligence Labs) | Composite indices | Expert | In use | 3 | 3 | Oct 2026 | Nov 2026 | |
| Vibe Code Bench 1.1 End-to-end web application development from natural-language specs. Built apps are tested against functional requirements. | Vals AI | Building whole projects | Expert | In use | Claude Fable 5.2 93.8% | 3 | 2 | Mar 2026 | Nov 2026 |
| Vibe Code Bench v1 Replaced in Mar 2026 | Not in cards | GPT-5.2 41.3% | Nov 2025 | Feb 2026 | |||||
| BigCodeBench-Hard Models solve 148 harder BigCodeBench Python tasks that call many libraries, chosen to match real programming work. Scored pass@1 on unit tests. | BigCode project | Function-level problems | University | Dropped | Claude 3.7 Sonnet 35.8% | 3 | 2 | Jul 2024 | Jun 2025 |
| DeepResearch Bench Agents write cited research reports for 100 PhD-level tasks across 22 fields. Judges score report quality against references and check citation accuracy. | University of Science and Technology of China and Metastone Technology | Deep research | Expert | Fading | Claude Opus 4.6 55.3% | 3 | 2 | Jun 2025 | May 2026 |
| MicroVQA Models answer expert-written multiple choice questions about biology microscopy images that test image understanding, hypothesis generation and experiment proposal. | Stanford University (James Burgess, Serena Yeung-Levy et al.) | General visual questions | Expert | Fading | GPT-5 59.4% | 3 | 2 | Mar 2025 | Sep 2026 |
| SEED-Bench-2 Multimodal models answer multiple-choice questions that cover comprehension of interleaved image and text inputs and image generation, in hierarchical capability levels. | Tencent AI Lab (ARC Lab) | General visual questions | Everyday | Not in cards | | Nov 2023 | Feb 2024 | ||
| SEED-Bench v1 Replaced in Nov 2023 | Tencent AI Lab and ARC Lab (Li et al.) | Fading | | 3 | 2 | Jul 2023 | Aug 2026 | ||
| SWE-sweep Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved. | Meta Superintelligence Labs (with Harvard, UW and Stanford) | Fixing real repositories | Research frontier | In use | GPT-6 Astra 8.90% | 3 | 2 | Oct 2026 | Nov 2026 |
| MM-IFEval Multimodal models follow image-grounded instructions with exact output constraints. Checks mix rule-based verification and a judge model. | MM-IFEngine team (Ding et al.) | Following precise instructions | Everyday | In use | GPT-4o 64.6% | 3 | 2 | Apr 2025 | Oct 2026 |
| FrontierScience-Research Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric. | OpenAI | Research science | Expert | Fading | | 3 | 2 | Dec 2025 | Jun 2026 |
| AVUT Models answer questions about videos where the audio carries the key information. The benchmark removes questions that text alone can answer. | Yang et al. (arXiv 2503.19951) | Video understanding | Everyday | Fading | | 3 | 2 | Mar 2025 | Sep 2026 |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | Center for AI Safety and Scale AI (HLE), Alibaba Tongyi Lab (VL subset) | General visual questions | Expert | Fading | o4-mini 16% | 3 | 2 | Aug 2025 | Aug 2026 |
| Omni-Cloze Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail. | Ma et al. (Omni-Captioner, arXiv 2510.12720) | Video captioning | Everyday | Fading | | 3 | 2 | Oct 2025 | Sep 2026 |
| HAE-RAE Bench 1.1 Models answer Korean questions on vocabulary, history, general knowledge and reading comprehension. It tests cultural and contextual knowledge that translated benchmarks miss. | HAE-RAE (Guijin Son et al.) | Knowledge in many languages | School | Fading | | 3 | 2 | Sep 2023 | Jul 2026 |
| HAE-RAE Bench 1.0 Replaced in Sep 2023 | Guijin Son and colleagues | Not in cards | May 2023 | ||||||
| KBank-MMLU (Upstage) Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house. | Upstage | Finance | Expert | Fading | | 3 | 2 | Jul 2025 | Jul 2026 |
| Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. | Vals AI | Law | Expert | In use | | 3 | 1 | Jun 2026 | Nov 2026 |
| HiPhO Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. | Shanghai AI Laboratory | Science olympiads | Competition | In use | | 3 | 1 | Sep 2025 | Aug 2026 |
| MobileWorld Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. | Tongyi Lab, Alibaba Group | Phones | Everyday | In use | GPT-5 55.3% | 3 | 1 | Dec 2025 | Aug 2026 |
| SALAD-Bench Models answer harmful questions organized in a three-level taxonomy, some with attack or defense changes and some multiple-choice. The MD-Judge evaluator labels responses as safe or unsafe. | Shanghai AI Laboratory (OpenSafetyLab) | Refusing harmful requests | Everyday | Fading | internlm-chat-20b 96.8% | 3 | 1 | Feb 2024 | Apr 2026 |
| ComplexFuncBench Models make function calls for 1,000 travel booking style samples that need several steps, user constraints, implicit parameter reasoning, long parameter values or 128K context. | Zhipu AI and Tsinghua University (THUDM) | Function calling | University | Dropped | Claude 3.5 Sonnet (October 2024) 61% | 3 | 1 | Jan 2025 | Jun 2025 |
| BrowseComp-Plus Deep research agents answer 830 BrowseComp questions by searching a fixed corpus of about 100,000 human-verified documents. | University of Waterloo and collaborators | Deep research | Expert | In use | GPT-5 70.1% | 3 | 1 | Aug 2025 | Oct 2026 |
| SpreadsheetBench 2 Agents complete 321 expert-annotated business spreadsheet workflows, such as financial modeling, debugging, and charting, in multi-sheet workbooks. Scoring is task accuracy. | Renmin University of China (RUCKBReasoning) | Documents and spreadsheets | Expert | In use | Claude Opus 4.6 34.9% | 3 | 1 | Jun 2026 | Nov 2026 |
| AMC Models solve problems from the AMC 8, 10, and 12 competitions. Gemini 1.5 used 250 problems from 2022 and 2023, scored by accuracy. | Mathematical Association of America | Competition problems with answers | Competition | Fading | | 3 | 1 | Feb 2024 | Dec 2025 |
| DeepMind Mathematics Models answer generated school-level math questions across algebra, arithmetic, calculus, probability and other modules. Answers are checked for an exact match. | DeepMind (Saxton et al.) | Word problems | School | Fading | Granite 4.1 30B 81.9% | 3 | 1 | Apr 2019 | Apr 2026 |
| SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. | UC San Diego + Stanford + UC Berkeley + Google DeepMind | Simulated robots | Everyday | In use | Praxis-1 80% | 3 | 1 | May 2024 | Nov 2026 |
| Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages. | Moonshot AI | Fixing real repositories | Expert | In use | Kimi K3.1 Pro 77.1% | 3 | 1 | Jun 2026 | Oct 2026 |
| Vistra Systems translate text in natural images across multiple language directions. | Google Research | Translation | University | Fading | | 3 | 1 | Sep 2024 | Jan 2026 |
| WorkSpaceBench Agent completes 388 tasks in realistic worker file workspaces (20k files) needing cross-file dependency reasoning. Scored against 7,399 rubrics. | SJTU + ByteDance (OpenDataBox) | Documents and spreadsheets | University | In use | | 3 | 1 | May 2026 | Aug 2026 |
| BankerToolBench v1.1 Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes. | Handshake AI Research | Finance | Expert | Not in cards | Claude Opus 5 32.6% | Oct 2026 | Jul 2026 | ||
| BankerToolBench 1.0 Replaced in Oct 2026 | In use | GPT-5.4 16% | 3 | Apr 2026 | Oct 2026 | ||||
| GSM-Symbolic Models solve grade-school word problems generated from symbolic templates of GSM8K questions, with names and numbers changed. P1 and P2 add one or two extra clauses. | Apple (Mirzadeh et al.) | Word problems | School | Fading | GPT-4o 94.9% | 3 | Oct 2024 | Apr 2026 | |
| RecreationBench Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result. | Qwen | Desktop apps | Expert | In use | GPT-6 Astra 58.1% | 3 | Sep 2026 | Sep 2026 | |
| BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. | Alibaba Tongyi Lab | Deep research | Expert | Fading | | 3 | Aug 2025 | Jun 2026 | |
| Management Consulting Tasks (OpenAI) Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading. | OpenAI | Running a business | Expert | In use | GPT-5.6 Sol 43.2% | 3 | Jul 2026 | Jul 2026 | |
| BigLaw Bench Models review legal documents and receive rubric scores for substantive accuracy and editing quality. | Harvey | Law | Research frontier | Fading | GPT-5.4 91% | 3 | Aug 2024 | Apr 2026 | |
| olmOCR-Bench OCR systems convert 1,403 PDFs to markdown. 7,010 unit tests check text presence and absence, reading order, tables and math. | Allen Institute for AI | Documents, charts and text in images | Everyday | In use | OXR-1.0 87.8% | 3 | Feb 2025 | Oct 2026 | |
| BrowseComp Long Context 128k Models answer BrowseComp questions from a 128k-token context of the needed web pages mixed with related distractor pages. Scored by accuracy. | OpenAI | Reasoning over long documents | Expert | Dropped | GPT-5.1 90% | 3 | Aug 2025 | Nov 2025 | |
| Hy-Backend 2.0 Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published. | Tencent Hy team | Fixing real repositories | Expert | In use | | 2 | Jul 2026 | Aug 2026 | |
| Hy-Backend v1 Replaced in Jul 2026 | Fading | | 1 | Apr 2026 | Apr 2026 | ||||
| Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task. | Tencent Hy team | Fixing real repositories | Expert | In use | | 3 | Apr 2026 | Aug 2026 | |
| Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. | Qwen | Knowledge in many languages | University | Dropped | | 3 | Feb 2024 | Sep 2024 | |
| Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. | Qwen Team, Alibaba Group | Reasoning in many languages | Everyday | Dropped | | 3 | Feb 2024 | Sep 2024 | |
| Kernel Bench L3 Models write fast GPU kernels for the 50 full model architectures in KernelBench Level 3. Scored on correctness and speedup over PyTorch. | Stanford Scaling Intelligence Lab | Speed and optimization | Expert | Fading | 3 | Dec 2024 | Jun 2026 | ||
| MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. | Center for AI Safety + Scale AI | Deception and alignment | Fading | | 2 | 21 | Mar 2025 | Oct 2026 | |
| ECLeKTic Models answer 384 closed-book questions, translated into 12 languages, about facts found in only one language's Wikipedia. Scoring checks transfer across languages. | Google Research and Google DeepMind | Knowledge in many languages | Everyday | Dropped | Claude Mythos Preview 56.3% | 2 | 10 | Feb 2025 | May 2026 |
| AgentDojo Agents do tool-use tasks while tools and data carry prompt injections. Attack success rate counts hijacked runs. | ETH Zurich SPY Lab + Invariant Labs | Jailbreaks and prompt injection | Everyday | Fading | Grok 4 Fast 0% | 2 | 8 | Jun 2024 | Oct 2026 |
| AutoLogi Models solve open-ended logic puzzles in English and Chinese. Generated verifier programs check each answer against the puzzle constraints. | Qwen Team, Alibaba Group and Fudan University | Abstract puzzles and logic | School | Dropped | Kimi K2 89.5% | 2 | 8 | Feb 2025 | Jul 2025 |
| Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. | University of Oregon NLP Group (Okapi) | Knowledge in many languages | University | Dropped | | 2 | 8 | Jul 2023 | Jul 2025 |
| TreeBench Models answer 405 visual questions about small targets in dense SA-1B images and give bounding boxes as evidence. Scored by answer accuracy, with box overlap as a second measure. | Wang et al. (TreeVGR authors) | Detection and captioning | University | Fading | | 2 | 7 | Jul 2025 | Jun 2026 |
| VSI-Bench Models watch egocentric videos of 288 indoor scenes and answer over 5,000 questions on object counts, distances, sizes, room layout and route planning. | New York University (VisionX, Saining Xie) with Stanford and Yale | Spatial reasoning and counting | Everyday | Fading | | 2 | 6 | Dec 2024 | Oct 2026 |
| LOFT Models solve retrieval, RAG, SQL, and many-shot learning tasks with whole corpora of up to one million tokens in context. Scored per task. | Google DeepMind | Finding facts in long inputs | Everyday | Dropped | | 2 | 6 | Jun 2024 | Jul 2026 |
| MTVQA-Test Models answer 6,778 expert-written questions about text in 2,116 images across 9 languages, such as Arabic, Japanese, and Thai. Scoring is accuracy. | ByteDance | Documents, charts and text in images | Everyday | Fading | | 2 | 5 | May 2024 | Aug 2026 |
| EvalPlus HumanEval+/MBPP+ with 80x more tests. Showed many 'passing' solutions were wrong. | UIUC | Function-level problems | School | Dropped | GLM-5 87% | 2 | 5 | May 2023 | Feb 2026 |
| MMVP Models answer 300 questions about 150 image pairs that CLIP embeds as near duplicates. A pair counts only when both answers are right. | New York University and UC Berkeley (Tong et al.) | Spatial reasoning and counting | Everyday | Fading | Kimi K2.5 87% | 2 | 5 | Jan 2024 | Mar 2026 |
| IndicGenBench Models summarize, translate, and answer questions across 29 Indic languages using human-translated versions of CrossSum, FLORES, XQuAD, and XorQA. Each task uses its standard metric. | Google Research | Reading in many languages | Everyday | Dropped | | 2 | 4 | Apr 2024 | Nov 2025 |
| MME-RealWorld Models answer human-annotated multiple choice questions about high-resolution real-world images in scenarios such as remote sensing, autonomous driving, monitoring and OCR. | MME team (Yi-Fan Zhang, Chaoyou Fu et al.) | General visual questions | Everyday | Dropped | | 2 | 4 | Aug 2024 | Nov 2025 |
| ReMI Models reason over several images per question in math, physics, logic, code, charts, and spatial or temporal tasks. Scored by accuracy. | Google DeepMind | General visual questions | School | Fading | | 2 | 4 | Jun 2024 | Jan 2026 |
| TallyQA Models answer open-ended counting questions about images, including complex ones that need object relationships or attributes. Scored by exact count accuracy. | Rochester Institute of Technology | Spatial reasoning and counting | Everyday | Dropped | | 2 | 4 | Oct 2018 | Apr 2025 |
| Spider 2.0 Models and agents write SQL for real enterprise databases with large schemas, across BigQuery, Snowflake, SQLite and dbt projects. Answers are checked by executing them. | XLANG Lab, The University of Hong Kong, with collaborators | Data science and SQL | Expert | Dropped | | 1 | Nov 2024 | Mar 2026 | |
| Spider 1.0 Replaced in Nov 2024 | Yale University (LILY lab, Tao Yu et al.) | University | Dropped | | 1 | 3 | Sep 2018 | Feb 2026 | |
| AetherCode Models write solutions to problems from premier programming contests such as IOI and ICPC. Expert-validated test suites judge each solution. | ByteDance Seed and collaborators (Wang et al.) | Competitive programming | Competition | Fading | o4-mini 35.5% | 2 | 3 | Aug 2025 | Jun 2026 |
| ANLI Models label premise and hypothesis pairs as entailment, neutral, or contradiction. People wrote the pairs over three rounds to fool strong models. | Facebook AI Research (Nie et al.) | Language reasoning | Everyday | Dropped | PaLM 2-L 73.1% | 2 | 3 | Oct 2019 | Aug 2024 |
| Ref-L4 Models locate the object described by a long referring expression in an image. The set has 45,341 annotations over 365 object categories, scored by box accuracy. | Chen et al. (Ref-L4 authors) | Detection and captioning | Everyday | Fading | GLM-4.5V 89.5% | 2 | 3 | Jun 2024 | Jan 2026 |
| Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. | Qwen Team, Alibaba Cloud | Function calling | Everyday | Dropped | | 2 | 3 | Aug 2023 | Feb 2024 |
| LiveSports-3K Video models give real-time commentary and answer questions on sports clips. An LLM judge scores the free-form commentary. | Chen et al. (LiveCC team, NUS Show Lab and ByteDance) | Video captioning | Everyday | Fading | | 2 | 3 | Apr 2025 | Jun 2026 |
| Clotho-AQA Models answer crowdsourced yes or no and single word questions about short environmental audio clips. Scored by accuracy. | Tampere University (Lipping, Sudarsanam et al.) | Audio understanding and voice | Everyday | Dropped | | 2 | 3 | Apr 2022 | Feb 2026 |
| IOI 2025 Model solves the 2025 International Olympiad in Informatics tasks. Scored with official subtask points (max 600). | IOI (problems) and NVIDIA (harness) | Competitive programming | Competition | Fading | | 2 | 3 | Aug 2025 | Jun 2026 |
| MBXP Models solve MBPP problems transpiled with their tests into more than 10 programming languages, such as Java, C++ and Go. Unit tests check each solution. | Amazon (Athiwaratkun et al.) | Function-level problems | School | Dropped | | 2 | 3 | Oct 2022 | Aug 2025 |
| VisualPuzzles Models solve visual puzzles that need algorithmic, analogical, deductive, inductive and spatial reasoning but little domain knowledge. Scored by accuracy. | Carnegie Mellon University (NeuLab) | Spatial reasoning and counting | Everyday | Fading | o4-mini 57% | 2 | 2 | Apr 2025 | Feb 2026 |
| VisFactor Models take 20 vision-centric subtests adapted from a human cognitive psychology test battery, covering perception primitives and visual memory. Scored by accuracy. | CUHK (ARISE Lab) | Spatial reasoning and counting | Everyday | In use | GPT-5.5 55.9% | 2 | 2 | Feb 2025 | Aug 2026 |
| MSEarth Models answer questions about figures from Earth science papers across the atmosphere, cryosphere, hydrosphere, lithosphere and biosphere. Labs report the multiple choice split. | Shanghai AI Lab (Xiangyu Zhao, Lei Bai et al.) | General visual questions | Expert | Fading | | 2 | 2 | May 2025 | Mar 2026 |
| MARS-Bench Models handle long multi-turn dialogues built from play-by-play sports commentary. Tasks test ultra multi-turn, interactive multi-turn and cross-turn reasoning. | ByteDance Seed and collaborators (Yang et al.) | Multi-turn conversation | Everyday | Fading | Claude 3.7 Sonnet 62.3% | 2 | 2 | May 2025 | Mar 2026 |
| OmniBench Models answer multiple choice questions that need an image, an audio clip and text together. Scored by accuracy. | Multimodal Art Projection (M-A-P) | Audio understanding and voice | Everyday | Dropped | | 2 | 2 | Sep 2024 | Oct 2025 |
| VideoReasonBench Video models watch a sequence of fine-grained operations on a partly hidden state and answer questions that recall, infer or predict that state. | Liu et al. (VideoReasonBench team) | Video understanding | Everyday | Fading | | 2 | 2 | May 2025 | Apr 2026 |
| Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. | Meta FAIR | General assistants | Everyday | Fading | GPT-5 42.1% | 2 | 2 | Sep 2025 | Oct 2026 |
| PhyX Models solve 3,000 image-based physics problems across six domains such as mechanics and optics. Versions include multiple choice and open-ended answers. | Hui Shen et al. | Science questions | University | Fading | o4-mini 45.8% | 2 | 2 | May 2025 | Aug 2026 |
| HRM8K-KSM Models solve the 1,428 problems of the Korean School Math subset of HRM8K. Problems come from Korean olympiad and competition exams. | HAE-RAE and OneLineAI (Hyunwoo Ko et al.) | Competition problems with answers | Competition | Fading | | 2 | 2 | Jan 2025 | Aug 2026 |
| CG-Bench Video models answer 12,129 questions about 1,219 long videos and must also find the clue segments that support each answer. | Chen et al. (CG-Bench team) | Long videos | Everyday | Fading | VITA 33% | 2 | 2 | Dec 2024 | Apr 2026 |
| ViSpeak-Bench Streaming video models must notice instructions given visually, such as a hand wave, and respond in seven subtasks of visual instruction feedback. | Fu et al. (ViSpeak team) | Video understanding | Everyday | Fading | GPT-4o 59.8% | 2 | 2 | Mar 2025 | Apr 2026 |
| VisualWebBench Models answer questions about web page screenshots across seven tasks, such as captioning, OCR, element grounding and action prediction. | Junpeng Liu, Yifan Song et al. | Finding things on screen | Everyday | Dropped | | 2 | 2 | Apr 2024 | Oct 2025 |
| AV-SpeakerBench Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together. | Nguyen et al. (arXiv 2512.02231) | Video understanding | Everyday | Fading | | 2 | 2 | Dec 2025 | Sep 2026 |
| DiscoX Models translate 200 expert texts between Chinese and English at discourse level, averaging over 1,700 tokens each. Metric-S, a reference-free judge, scores accuracy, fluency and appropriateness, and the overall score is their sum. | ByteDance Seed and collaborators (Zhao et al.) | Translation | Expert | Fading | GPT-5 76.7 | 2 | 2 | Nov 2025 | Mar 2026 |
| CorpusQA 1M Models answer computation-heavy questions whose evidence is spread across hundreds of finance, education, and real estate documents in a 1M-token context. Scored by accuracy. | Tongyi Zhiwen (Alibaba) | Reasoning over long documents | University | Fading | | 2 | 2 | Dec 2025 | Apr 2026 |
| AndroidControl Agents predict the next action on Android app screens from either a high-level goal or a low-level step instruction. Scored by step accuracy against human demonstrations. | Google DeepMind (Wei Li, William Bishop et al.) | Phones | Everyday | Dropped | | 2 | 2 | Jun 2024 | Oct 2025 |
| VATEX ZH Models write Chinese captions for short videos of human activities from the VATEX dataset. Captions are scored with CIDEr against human references. | UC Santa Barbara (UCSB NLP Group) | Video captioning | Everyday | Dropped | | 2 | 2 | Apr 2019 | Jul 2025 |
| YouCook2 Models describe the procedure steps in YouTube cooking videos. Scored by CIDEr against human step captions. | University of Michigan | Video captioning | Everyday | Dropped | | 2 | 2 | Mar 2017 | Jul 2025 |
| CanItEdit Models edit a given block of code to follow an instruction, such as adding a feature, fixing a described bug or changing the approach. Hidden tests check the edited code. | Northeastern University Programming Research Lab (Cassano et al.) | Function-level problems | University | Dropped | | 2 | 2 | Dec 2023 | Aug 2025 |
| CMO 2025 Models write full proofs for the six problems of the 2025 China Mathematical Olympiad. Graders score proofs out of 126 points. | Chinese Mathematical Society | Written proofs | Competition | Fading | | 2 | 2 | Nov 2025 | Feb 2026 |
| DeepShop Web agents answer complex shopping queries on live sites with product attributes, search filters and sorting preferences. An LLM judge checks whether results meet each requirement. | Lyu et al. (University of Amsterdam and collaborators) | Web browsing | Everyday | Fading | | 2 | 2 | Jun 2025 | Apr 2026 |
| ICPC World Finals 2025 Models solve the 12 problems of the 2025 ICPC World Finals under contest time and attempt limits. Scored by problems solved. | ICPC Foundation | Competitive programming | Competition | Fading | | 2 | 2 | Sep 2025 | Mar 2026 |
| XpertBench Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics. | ByteDance Seed | Tasks from real jobs | Expert | Fading | | 2 | 2 | Dec 2025 | Apr 2026 |
| DeepConsult Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer. | You.com | Deep research | Expert | Fading | | 2 | 2 | May 2025 | Mar 2026 |
| DPG-Bench Text-to-image models draw 1,000 dense prompts with many objects, attributes and relations. A VQA model checks each element of the prompt graph in the image. | Tencent (ELLA authors, Hu et al.) | Image generation and editing | Everyday | Fading | | 2 | 2 | Mar 2024 | Jan 2026 |
| FlenQA Models answer reasoning questions whose key facts are padded with filler text to different lengths. Accuracy shows how input length affects reasoning. | Mosh Levy, Alon Jacoby, Yoav Goldberg | Reasoning over long documents | Everyday | Dropped | | 2 | 2 | Feb 2024 | Apr 2025 |
| Harvey LAB-AA v1.0 Artificial Analysis runs agents on Harvey's Legal Agent Benchmark tasks in an offline sandbox, where they write legal work products such as memos and redlines from case documents. Judges grade each deliverable against a task rubric, and the headline is the share of rubric criteria passed. | Artificial Analysis | Law | Expert | In use | Kimi K3 94.6% | 2 | 2 | Aug 2026 | |
| KGC-Safety (LG) Models answer risky prompts set in Korean social and cultural contexts. A Safe Rate is scored over four domains: universal human values, social safety, Korean sensitivity and future risk. | LG AI Research | Refusing harmful requests | Everyday | Fading | | 2 | 2 | Jan 2026 | Aug 2026 |
| Ko-AIME 2025 (Upstage) Models solve AIME 2025 competition problems in Korean. Upstage built the set in house. | Upstage | Competition problems with answers | Competition | Fading | | 2 | 2 | Jan 2026 | Jul 2026 |
| LongCat safety evaluation (Meituan) Models answer private test queries in four risk categories: harmful content, criminal activity, misinformation and privacy. The score is the share of safe responses in each category. | Meituan LongCat Team | Refusing harmful requests | Everyday | Dropped | | 2 | 2 | Sep 2025 | Sep 2025 |
| LongCodeBench Coding model answers repository QA and fixes GitHub issues with context from 32K to 1M tokens. Accuracy and resolve rate. | Panasonic AI Research + Sapienza + Stanford | Reasoning over long documents | Expert | Fading | | 2 | 2 | May 2025 | Dec 2025 |
| LPFQA Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains. | ByteDance Seed and collaborators (Zhu et al.) | Expert-written questions | Expert | Fading | | 2 | 2 | Nov 2025 | Mar 2026 |
| Matbench Models predict properties of inorganic materials in 13 supervised tasks built from DFT and experimental data. Scored with error or classification metrics per task. | Materials Project (Dunn, Wang, Ganose, Dopp, Jain) | Research science | Expert | Fading | | 2 | 2 | May 2020 | Mar 2026 |
| Meeseeks Models follow complex instructions over several turns and get feedback on failed requirements after each turn. Scored by how many requirements they meet. | Meituan | Multi-turn conversation | Everyday | Dropped | | 2 | 2 | Apr 2025 | Oct 2025 |
| MMMB Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. | Alibaba AIDC-AI (Parrot authors) | Knowledge in many languages | School | In use | LFM2.5-VL-3B 83% | 2 | 2 | Jun 2024 | Aug 2026 |
| OMEGA Models solve generated competition-style math problems that test exploratory, compositional and transformative generalization beyond training patterns. Answers are checked automatically. | UC Berkeley, Ai2 and University of Washington (Sun et al.) | Competition problems with answers | Competition | Fading | | 2 | 2 | Jun 2025 | Dec 2025 |
| OpenAudioBench Models answer spoken questions in five sets: Reasoning QA, Llama Questions, Web Questions, TriviaQA and AlpacaEval. Knowledge sets use accuracy and the open sets use a model score. | Baichuan | Audio understanding and voice | Everyday | Dropped | | 2 | 2 | Jan 2025 | Oct 2025 |
| SMolInstruct Models solve 14 chemistry tasks such as name conversion, property prediction, molecule captioning, generation and reaction prediction. Each task uses its own metric on the test split. | The Ohio State University (LlaSMol team) | Genomics and life sciences | Expert | Fading | | 2 | 2 | Feb 2024 | Mar 2026 |
| TydiQA Models answer information-seeking questions from Wikipedia articles in 11 typologically diverse languages, across 204K question-answer pairs. Scored by F1. | Google Research | Reading in many languages | Everyday | Dropped | | 2 | 2 | Mar 2020 | Apr 2025 |
| Kitab Models list books by a given author that meet extra constraints, such as publication dates or title words. Scored by precision and constraint satisfaction. | Microsoft Research | Long-form factuality | Everyday | Dropped | 2 | 2 | Oct 2023 | Apr 2025 | |
| MemoryBench (Meta) Models recall key facts from multi-turn conversations of 64K to 100K tokens that include distractor chats and turns that conflict with or update the facts. An LLM judge scores answers on weighted rubrics. | Meta (Meta Superintelligence Labs) | Finding facts in long inputs | Everyday | In use | 2 | 2 | Oct 2026 | Oct 2026 | |
| Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. | Vals AI | Finance | Expert | In use | | 2 | 1 | Sep 2026 | Nov 2026 |
| ProfBench Model writes expert-level reports in physics, chemistry, finance and consulting. Calibrated LLM judges grade against PhD/MBA-written rubrics. | NVIDIA | Tasks from real jobs | Expert | In use | Claude Opus 5 66.2% | 2 | 1 | Oct 2025 | Aug 2026 |
| Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. | Vals AI | Composite work indices | Expert | In use | | 2 | 1 | Oct 2026 | Nov 2026 |
| CFBench Models follow 1,000 real-world instructions with embedded constraints, across more than 200 scenarios and 50 NLP tasks. A judge checks each constraint and the overall requirement. | Baichuan and Peking University | Following precise instructions | Everyday | Dropped | o1-preview 71.8% | 2 | 1 | Aug 2024 | Sep 2025 |
| RefusalBench (Nous Research) Models get requests in scenarios that closed and open models commonly refuse. The score is the percent of questions the model answers, averaged over 5 trials. | Nous Research | Deception and alignment | Everyday | Fading | Hermes 4 405B 57.1% | 2 | 1 | Aug 2025 | Dec 2025 |
| ComplexConstraints Models answer 75 expert-written professional prompts whose constraints depend on each other, fire conditionally or must be inferred. An LLM judge grades 1,559 rubric criteria one at a time, and a task passes only if all criteria pass. | Surge AI | Following precise instructions | University | In use | GPT-6 Astra 70.8% | 2 | 1 | Jun 2026 | Nov 2026 |
| CountQA Models count objects in over 1,500 real-world photos with dense clutter and occlusion. Scoring is accuracy on the count. | Spatial reasoning and counting | Everyday | In use | | 2 | 1 | Aug 2025 | Aug 2026 | |
| MMAU-Pro Models answer 5,305 expert-written questions about speech, sounds, music and their mixes, covering 49 listening skills. Some clips run up to 10 minutes. | University of Maryland (GAMMA lab) and collaborators | Audio understanding and voice | University | Fading | | 2 | 1 | Aug 2025 | Aug 2026 |
| $OneMillion-Bench Agent solves 400 expert-curated tasks in law, finance, industry, healthcare and science. Rubric grading of accuracy, logic, feasibility and compliance. | Humanlaya + BIGAI + xbench | Tasks from real jobs | Expert | In use | Claude Fable 5.1 42.1% | 2 | 1 | Mar 2026 | Sep 2026 |
| PMPP-Hard Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. | Sinatras, sponsored by Prime Intellect | GPU kernels | Research frontier | In use | Kimi K3 0.71 | 2 | 1 | Jul 2026 | Nov 2026 |
| HorizonMath Models attack 113 mostly unsolved problems in eight areas of mathematics and the mathematical sciences. A program checks each proposed closed form, construction or bound. | University of Oxford, Harvard University, Princeton University and others (Wang et al.) | Research mathematics | Research frontier | In use | GPT-5.6 Sol 11.5% | 2 | 1 | Mar 2026 | Aug 2026 |
| BABE Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. | ByteDance Seed and collaborators (Zhou et al.) | Genomics and life sciences | Expert | Fading | GPT-5.1 52.3% | 2 | 1 | Feb 2026 | Apr 2026 |
| HR-Bench Models answer multiple-choice questions about fine details in 4K and 8K resolution images. Results are reported separately for the 4K and 8K sets. | Wuhan University and collaborators (Wang et al.) | General visual questions | Everyday | Fading | | 2 | 1 | Aug 2024 | May 2026 |
| MultiLoKo Locally relevant knowledge questions in 31 languages, plus translations. Scored by exact match. | Meta FAIR | Knowledge in many languages | Everyday | Fading | | 2 | 1 | Apr 2025 | Sep 2026 |
| OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. | Nanjing University (NJU-LINK Lab) and others | Video understanding | University | Fading | | 2 | 1 | Oct 2025 | Sep 2026 |
| JointAVBench Models answer questions about videos that cannot be answered from vision or audio alone. Questions cover speech and sound events over several scene spans. | Chao et al. (arXiv 2512.12772) | Video understanding | Everyday | Fading | | 2 | 1 | Dec 2025 | Sep 2026 |
| MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code. | Xiaomi MiMo | Web and app front ends | Expert | In use | GPT-6 Astra 82.2% | 2 | 1 | Sep 2026 | Sep 2026 |
| FATE-H Provers write Lean proofs for 100 abstract and commutative algebra problems at honors course or graduate exam level. The Lean compiler checks each proof. | Jiang et al. (FATE team) | Formal proofs | Expert | Fading | o3 3.00% | 2 | 1 | Nov 2025 | Jul 2026 |
| FATE-X Provers write Lean proofs for 100 algebra problems at or beyond PhD qualifying exam level, some beyond what Mathlib covers. The Lean compiler checks each proof. | Jiang et al. (FATE team) | Formal proofs | Research frontier | Fading | o3 0% | 2 | 1 | Nov 2025 | Jul 2026 |
| MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details. | Xiaomi MiMo | Fixing real repositories | Expert | In use | Claude Opus 5 68.6% | 2 | 1 | Sep 2026 | Sep 2026 |
| RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. | NVIDIA | Simulated robots | Everyday | In use | Praxis-1 46.2% | 2 | 1 | Apr 2026 | Nov 2026 |
| RWSearch Agents search the live web to answer 200 expert-written Chinese questions that need many retrieval steps, filtering and arithmetic. Each question has one objective answer. | AGI-Eval (Meituan) | Deep research | Expert | Fading | GPT-5.2 82% | 2 | 1 | Jan 2026 | Jul 2026 |
| MMEB-V3 Omni-modality embedding models are scored on 190 tasks across text, image, video, audio, visual documents and agent-centric retrieval. | University of Waterloo (TIGER-Lab) and collaborators | General visual questions | Everyday | Not in cards | | Apr 2026 | Jan 2026 | ||
| MMEB-V2 Replaced in Apr 2026 | University of Waterloo (TIGER-Lab) + Salesforce Research | Fading | | 2 | 1 | May 2025 | Oct 2026 | ||
| MMEB v1 Replaced in May 2025 | University of Waterloo (TIGER-Lab) + Salesforce Research | Not in cards | | Oct 2024 | Jan 2026 | ||||
| CombiBench Provers solve 100 combinatorics problems formalized in Lean 4, from middle school to IMO and university level. Lean checks each proof. | Moonshot AI and Project Numina (Liu et al.) | Formal proofs | Competition | Fading | | 2 | 1 | May 2025 | Dec 2025 |
| MedXpertQA Models answer 4,460 expert-level medical exam questions across 17 specialties, in a text subset and an image subset. Scoring is multiple-choice accuracy. | Tsinghua University (TsinghuaC3I) | Medicine | Expert | Fading | | 2 | 1 | Jan 2025 | Jun 2026 |
| TheoremQA Models solve 800 university-level questions in math, physics, EE and CS, and finance that each need a specific theorem. Scored by answer accuracy. | University of Waterloo (TIGER-AI-Lab) | Science questions | University | Dropped | | 2 | 1 | May 2023 | Sep 2024 |
| MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. | Xiaomi MiMo | Finding and exploiting real bugs | Expert | In use | MiMo-V2.6 Pro 81.7% | 2 | 1 | Sep 2026 | Sep 2026 |
| Hungarian National High-School Exam Models answer the 33 questions of the 2023 Hungarian national high school mathematics finals. Answers are graded by hand with the official rubric. | Keiran Paster (exam by the Hungarian Educational Authority) | Word problems | School | Dropped | Grok-1 59% | 2 | 1 | Nov 2023 | Jan 2024 |
| PresentBench Models generate full slide decks from background materials for 238 real-world cases. A judge scores each deck against about 54 hand-written checklist items. | PresentBench team | Documents and spreadsheets | University | In use | | 2 | 1 | Mar 2026 | Sep 2026 |
| RACE-M Models read passages from English exams for Chinese middle school students and answer multiple-choice questions. Scored by accuracy. | Carnegie Mellon University | Reading comprehension | School | Dropped | PaLM 2-L 77% | 2 | 1 | Apr 2017 | Dec 2024 |
| SpreadsheetBench Verified Models solve 400 real spreadsheet manipulation questions from Excel forums, an expert annotated subset of SpreadsheetBench. Each answer runs against several test spreadsheets. | Renmin University of China (RUCKBReasoning) with Shortcut.AI | Documents and spreadsheets | University | Fading | | 2 | 1 | Dec 2025 | Jun 2026 |
| AA-AnalystAgent Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right. | Artificial Analysis | Running a business | Expert | In use | GPT-6.2 Astra 62.5% | 2 | 1 | Aug 2026 | Nov 2026 |
| ASIMOV 2.0 Models spot physical hazards in text and generated videos from real injury reports, time safety interventions in video, and follow robot payload and handling constraints. | Google DeepMind | Embodied reasoning | Fading | | 2 | 1 | Sep 2025 | Oct 2026 | |
| ASIMOV v1 1.0 Replaced in Sep 2025 | Everyday | Not in cards | Mar 2025 | ||||||
| Biology-Instructions Models answer multi-omics sequence tasks on DNA, RNA, protein and multi-molecule inputs, such as property prediction and interaction tasks. Scored per task with task metrics. | Biology-Instructions team (hhnqqq et al.) | Genomics and life sciences | Expert | Fading | | 2 | 1 | Dec 2024 | Sep 2026 |
| Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. | Thinking Machines Lab | Deception and alignment | Expert | In use | | 2 | 1 | Nov 2026 | Nov 2026 |
| Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. | Thinking Machines Lab | Deception and alignment | Expert | In use | | 2 | 1 | Nov 2026 | Nov 2026 |
| DiligenceBench 150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness. | Thoughtful Lab and Paper Instruments | Finance | Expert | In use | | 2 | 1 | Jul 2026 | Nov 2026 |
| Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. | OpenAI | Building whole projects | Expert | Fading | GPT-6 Cyber 83% | 2 | 1 | Apr 2026 | Nov 2026 |
| GQA Models answer compositional questions about real-world images, generated from scene graphs. Scored by exact-match accuracy, often on the balanced test-dev split. | Stanford University (Hudson and Manning) | General visual questions | Everyday | Fading | | 2 | 1 | Feb 2019 | Aug 2026 |
| Ko-GDPval Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format. | Upstage | Tasks from real jobs | Expert | In use | | 2 | 1 | Jul 2026 | Aug 2026 |
| Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. | Thinking Machines Lab | Deception and alignment | Expert | In use | | 2 | 1 | Nov 2026 | Nov 2026 |
| MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. | Google DeepMind and Artificial Analysis | Tasks from real jobs | Expert | In use | | 2 | 1 | Nov 2026 | Nov 2026 |
| Mol-Instructions Models answer molecule-oriented, protein-oriented and biomolecular text instructions, such as reaction prediction, molecule description and protein function questions. Scored per task with task metrics. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Fading | | 2 | 1 | Jun 2023 | Sep 2026 |
| SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. | Shanghai AI Lab (open-sciencelab) | Research science | Expert | Fading | | 2 | 1 | Sep 2025 | Sep 2026 |
| SpeechRole Speech role-playing agents talk in character across 98 roles. A model judge scores interaction ability, speech expressiveness and role fidelity from the generated speech. | Jiang et al. (arXiv 2508.02013) | Speech recognition and translation | Everyday | Fading | | 2 | 1 | Aug 2025 | Apr 2026 |
| ForecastBench Models give probabilities for 500 auto-generated questions per round about future events, drawn from prediction markets and time-series datasets. Forecasts are scored with difficulty-adjusted Brier scores once the questions resolve. | Forecasting Research Institute | Real-world events | Expert | In use | GPT-5.5 0.11 | 2 | Sep 2024 | Sep 2026 | |
| APEX-Accounting Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria. | Mercor with Ramp | Simulated companies | Expert | In use | | 2 | Jul 2026 | Nov 2026 | |
| PRBench Model answers 1,100 expert-written law and finance questions. LLM judge grades against weighted expert rubrics of 10-30 criteria each. | Scale AI | Law | Expert | In use | | 2 | Nov 2025 | Nov 2026 | |
| Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. | Vals AI | Composite indices | Expert | In use | GPT-6.2 Astra 68.4% | 2 | Oct 2026 | Nov 2026 | |
| Prophet Arena Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return. | UChicago SIGMA Lab with Kalshi | Real-world events | Expert | In use | Claude Opus 5.5 86.2 | 2 | Oct 2025 | Oct 2026 | |
| FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. | AGI-Eval (Meituan) | Documents and spreadsheets | University | In use | GPT-5.5 77.8% | 2 | Jun 2026 | Nov 2026 | |
| LiveClawBench Agents act as OpenClaw personal assistants on 134 multi-step tasks across 10 app domains backed by stateful mock services. A run passes when its reward exceeds 0.8. | Mosi-AI (Xiang Long, Yehui Tang and collaborators) | General assistants | Expert | In use | Kimi K2.7 Code 76% | 2 | Mar 2026 | Oct 2026 | |
| FrontierFinance Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics. | Samaya AI | Finance | Expert | In use | Claude Fable 5.1 55.9% | 2 | Jul 2026 | Oct 2026 | |
| CL-bench Life Models solve 405 tasks that require learning from messy real-life context such as group chats. A model judge checks rubrics to give a solve rate. | Tencent Hunyuan and Fudan University | Learning from the context | Everyday | Fading | GPT-5.5 22.2% | 2 | Apr 2026 | Jul 2026 | |
| T-Eval Models use tools step by step, and each sub-process is scored separately: instruction following, planning, reasoning, retrieval, understanding, and review. | Shanghai AI Laboratory (OpenCompass) | Function calling | Everyday | Dropped | GPT-4 Turbo 86.4% | 2 | Dec 2023 | Mar 2025 | |
| CMT-Benchmark Models solve 50 research-level condensed matter theory problems written and checked by a panel of expert physicists. Answers are graded automatically. | International panel of condensed matter researchers (Pan, Roggeveen, Berg, Carrasquilla, Chowdhury and others) | Research science | Research frontier | Fading | GPT-5 30% | 2 | Oct 2025 | Jul 2026 | |
| IFStruct Models generate JSON or YAML that must follow a requested schema and phrasing constraints, without constrained decoding. A response passes only if every structural constraint holds. | Liquid AI | Following precise instructions | Everyday | In use | | 2 | Jun 2026 | Oct 2026 | |
| BigFinanceBench Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference. | Rogo Technologies | Finance | Expert | In use | Claude Opus 4.7 58.8% | 2 | Jun 2026 | Jul 2026 | |
| EVA-Bench Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes. | ServiceNow | Speech recognition and translation | Everyday | In use | Grok Voice 3.0 0.74 | 2 | May 2026 | Oct 2026 | |
| LOCA-Bench Agentic tasks whose environment state is scaled to grow context length while task semantics stay fixed. Success rate versus context length. | HKUST NLP | Reasoning over long documents | University | In use | GPT-5.2 21.3% | 2 | Feb 2026 | Oct 2026 | |
| Dense 200 (bbox) Model detects and boxes every object in 200 densely annotated images (109 categories, about 91 boxes each). Box F1 at IoU thresholds. | IDEA Research (Rex-Omni) | Detection and captioning | Everyday | In use | | 2 | Oct 2025 | Oct 2026 | |
| SURDS Models answer spatial questions about objects in nuScenes driving camera images across six single-object and multi-object tasks. Scoring is task accuracy. | Xianda Guo et al. (Wuhan University and partners) | Spatial reasoning and counting | Everyday | In use | | 2 | Nov 2024 | Aug 2026 | |
| E-Bench (Tencent) Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published. | Tencent Hy team | Documents and spreadsheets | Expert | In use | GPT-5.6 Sol 80.6% | 2 | Jul 2026 | Aug 2026 | |
| QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. | Qwen Team, Alibaba Group | Web and app front ends | University | Fading | | 2 | May 2026 | Jun 2026 | |
| MMSearch-Plus Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key. | Authors of arXiv 2508.21475 | Deep research | Expert | Fading | | 2 | Aug 2025 | Jun 2026 | |
| QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. | Alibaba Qwen | Fixing real repositories | Expert | In use | Claude Fable 5 86.3% | 2 | Aug 2026 | ||
| QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. | Qwen Team, Alibaba Group | Web and app front ends | University | Fading | Claude Opus 4.6 1,617 | 2 | May 2026 | Jun 2026 | |
| QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. | Qwen Team, Alibaba Group | Interactive world models | Fading | | 2 | May 2026 | Jun 2026 | ||
| FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. | Meta (Meta Superintelligence Labs) | General assistants | Everyday | In use | | 2 | Oct 2026 | Oct 2026 | |
| Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. | Google DeepMind | Embodied reasoning | Everyday | Fading | | 2 | Apr 2026 | Jul 2026 | |
| MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE. | Xiaomi MiMo | Fixing real repositories | Expert | Fading | Claude Opus 4.6 77.1% | 2 | Apr 2026 | Apr 2026 | |
| Research-Eval Search-augmented models answer 374 grounded multi-hop questions that need sources across many kinds of websites. An LLM judge checks each answer against a checklist of requirements. | Reka AI | Deep research | Expert | Dropped | Grok 4 Fast 66% | 2 | Aug 2025 | Nov 2025 | |
| X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. | xAI | Deep research | Expert | Dropped | Grok 4 Fast 58% | 2 | Sep 2025 | Nov 2025 | |
| tau-bench Retail Agents help simulated retail customers through tools while following store policies on the original tasks. The score is pass^1 as a percentage. | Sierra | Customer service | Everyday | Dropped | gpt-oss-120b 67.8% | 2 | Jun 2024 | Aug 2025 | |
| Code Interpreter Benchmark Math Models use a Python interpreter to solve math problems. The score is execution accuracy on the Math task set. | Qwen | Function calling | School | Dropped | | 2 | Sep 2023 | Feb 2024 | |
| Decision Index Public v0.3 Decision models return probabilities over labeled options in public tasks covering knowledge, language, retrieval, tools, and arts. A chance-corrected weighted aggregate combines the task scores. | Hugging Face Decision Index community | Composite indices | University | In use | Granite Switch 4.1 30B 51.3 | 2 | Oct 2026 | ||
| GDPval v2 (OpenAI) Multistep assignments drawn from real projects in consulting, finance and design, given with the files and messages the team had when the work was assigned. OpenAI has not published the task count or grading. | OpenAI | Tasks from real jobs | Expert | In use | GPT-6.2 Astra 50.4% | 2 | Nov 2026 | Nov 2026 | |
| GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. | OpenAI | Genomics and life sciences | Expert | Fading | GPT-5.5 33.2% | 2 | Apr 2026 | Apr 2026 | |
| Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published. | Tencent Hy team | Building whole projects | Expert | In use | | 1 | Aug 2026 | Aug 2026 | |
| Hy-CompanyBench v1 Replaced in Aug 2026 | In use | | 1 | Jul 2026 | Jul 2026 | ||||
| Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. | Tencent Hy team | Finance | Expert | In use | | 1 | Aug 2026 | Aug 2026 | |
| Hy-FinModelBench v1 Replaced in Aug 2026 | In use | | 1 | Jul 2026 | Jul 2026 | ||||
| IndiVibe Models answer 110 source prompts in chat, STEM, math and coding, translated into 22 Indian languages in native and romanized script. A judge model compares answers pairwise. | Sarvam AI | Chat and instructions in many languages | Everyday | Fading | Sarvam 105B 90% | 2 | Mar 2026 | Mar 2026 | |
| KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages. | Kuaishou (KwaiKAT team) | Fixing real repositories | University | In use | | 2 | Jul 2026 | Jul 2026 | |
| LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. | Toyota Research Institute | Simulated robots | Everyday | Fading | Foundry-Qwen3VLA-2.1B 72.4% | 2 | May 2025 | Apr 2026 | |
| LifeVal Assistants carry out everyday personal tasks rather than advise on them: finding childcare, booking a car inspection, ordering groceries on a budget, handling government paperwork. Tasks come from time-use surveys and from about 100 people who recorded tasks they wanted an assistant to handle. Score is average reward across tasks. | OpenAI | Web browsing | Everyday | In use | GPT-6.2 Astra 86.1% | 2 | Nov 2026 | Nov 2026 | |
| PatchBench (OpenAI) Models patch security flaws. Automated tests and model-based review check that each patch fixes the flaw, keeps the intended behavior and is ready to merge. | OpenAI | Secure code and review | Expert | In use | GPT-6.2 Astra 79.7% | 2 | Nov 2026 | Nov 2026 | |
| PhysicsFinals Models answer 61 undergraduate physics final exam problems written by physics professors. A professor grades the answers. | Google DeepMind | Science questions | University | Dropped | | 2 | Feb 2024 | May 2024 | |
| SecureSWE (OpenAI) Models write functional and secure code for security-sensitive components, such as serving files or handling database queries. OpenAI has not published the task count or grading. | OpenAI | Secure code and review | Expert | In use | GPT-6.2 Astra 65.7% | 2 | Nov 2026 | Nov 2026 | |
| Speech-to-Speech Index Artificial Analysis's combined score for speech-to-speech models. | Artificial Analysis | Speech recognition and translation | Research frontier | In use | Grok Voice 3.0 86.9 | 2 | Jun 2026 | Oct 2026 | |
| Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. | Google DeepMind | Embodied reasoning | Everyday | Fading | | 2 | Sep 2025 | Jul 2026 | |
| XCOPA Models pick the more plausible cause or effect of a premise from two options, in 11 languages. Scoring is accuracy. | Cambridge Language Technology Lab (Ponti et al.) | Reasoning in many languages | Everyday | Dropped | PaLM 2-L 94.4% | 2 | May 2020 | Sep 2025 | |
| ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. | OpenAI | Lab protocols | Expert | In use | GPT-6 Cyber 47.2% | 1 | 37 | Dec 2024 | Nov 2026 |
| VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. | SecureBio | Lab protocols | Expert | System cards only | Claude Opus 5.5 60.5% | 1 | Sep 2026 | Oct 2026 | |
| Virology Capabilities Test (VCT) v1 Replaced in Sep 2026 | SecureBio and Center for AI Safety | Hazardous knowledge | Fading | Grok 4.6 67.4% | 1 | 35 | Feb 2025 | Oct 2026 | |
| HealthBench Consensus Models answer HealthBench conversations graded only on criteria that several physicians agreed on. | OpenAI | Medicine | Expert | Dropped | GPT-5.4 96.6% | 1 | 25 | May 2025 | Oct 2026 |
| Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. | OpenAI | Capture-the-flag challenges | Expert | Fading | GPT-5.5 96.3% | 1 | 22 | Aug 2024 | May 2026 |
| Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. | Stanford | Capture-the-flag challenges | Expert | Fading | | 1 | 19 | Aug 2024 | Jul 2026 |
| LABBench2 ProtocolQA2 Models find and fix errors planted in published lab protocols, answering in open response. The ProtocolQA2 subtask of LABBench2. | Edison Scientific (formerly FutureHouse) | Genomics and life sciences | Expert | Not in cards | | Feb 2026 | Feb 2026 | ||
| LAB-Bench ProtocolQA v1 Replaced in Feb 2026 | FutureHouse | Lab protocols | Fading | Claude Opus 4.5 90.7% | 1 | 19 | Jul 2024 | Oct 2026 | |
| CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. | Irregular | Capture-the-flag challenges | Expert | In use | | 1 | 19 | Dec 2025 | Nov 2026 |
| LABBench2 FigQA2 Models read scientific figures and answer in open response. Served as images or as PDFs. The FigQA2 subtask of LABBench2. | Edison Scientific (formerly FutureHouse) | Genomics and life sciences | Expert | Not in cards | GPT-5.2 42.6% | Feb 2026 | Feb 2026 | ||
| LAB-Bench FigQA v1 Replaced in Feb 2026 | FutureHouse | Fading | Claude Opus 4.8 80.4% | 1 | 15 | Jul 2024 | Jun 2026 | ||
| LAB-Bench CloningScenarios Models answer multiple-choice questions about multi-step molecular cloning workflows. CloningScenarios subset of LAB-Bench. | FutureHouse | Lab protocols | Expert | Fading | | 1 | 15 | Jul 2024 | Apr 2026 |
| Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. | OpenAI | Deception and alignment | Everyday | In use | GPT-6 Cyber 0.33% | 1 | 14 | Aug 2025 | Nov 2026 |
| Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. | Cohere Labs | Knowledge in many languages | University | Dropped | Claude Fable 5.2 96% | 1 | 13 | Dec 2024 | Nov 2026 |
| SimpleQA Hallucination Rate Models answer 4,326 short fact-seeking questions. This metric is the share of questions answered incorrectly, so declining to answer is not penalized. | OpenAI | Hallucination rates | Dropped | ChatGPT agent 7.90% | 1 | 12 | Oct 2024 | Oct 2025 | |
| Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. | OpenAI | Deception and alignment | Expert | In use | GPT-6 Astra 0% | 1 | 12 | Sep 2026 | Nov 2026 |
| Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. | OpenAI | Deception and alignment | Expert | In use | GPT-6 Luna 11.8% | 1 | 12 | Sep 2026 | Nov 2026 |
| QuAC Models answer free-form questions in information-seeking dialogs about a hidden Wikipedia section, across 100K questions. Scored by word-level F1. | Allen Institute for AI, University of Washington, Stanford, and UMass Amherst | Reading comprehension | Everyday | Dropped | | 1 | 12 | Aug 2018 | Nov 2024 |
| MILU Multi-task Indic Language Understanding: culturally grounded multiple-choice questions in 11 Indic languages. | AI4Bharat | Knowledge in many languages | University | Dropped | Claude Fable 5.2 94.1% | 1 | 10 | Nov 2024 | Nov 2026 |
| MakeMeSay A manipulator model tries to make a GPT-4o manipulatee say a codeword without suspicion. Scored as manipulator win rate. | OpenAI | Dangerous capabilities | Everyday | Fading | GPT-4.5 72% | 1 | 10 | Sep 2023 | Apr 2026 |
| Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. | Apollo Research | Deception and alignment | Expert | In use | Claude Sonnet 4.5 0% | 1 | 10 | Nov 2026 | |
| Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. | OpenAI | Deception and alignment | Expert | In use | GPT-6 Astra 0% | 1 | 9 | Sep 2026 | Nov 2026 |
| Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. | Anthropic | Deception and alignment | Expert | In use | Claude Sonnet 4.5 0% | 1 | 9 | Jun 2025 | Oct 2026 |
| MLogiQA Models answer LogiQA logical reasoning questions translated into 10 languages, 80 per language. Scored by accuracy. | Tongyi Lab, Alibaba Group | Reasoning in many languages | University | Dropped | | 1 | 8 | Nov 2024 | Sep 2025 |
| WMDP Biology Models answer 1,273 multiple-choice questions that proxy hazardous biosecurity knowledge. Biology split of WMDP. | CAIS + Scale AI et al. | Hazardous knowledge | Expert | Fading | Grok 4.20 91% | 1 | 8 | Mar 2024 | Apr 2026 |
| General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. | xAI (SpaceXAI) | Refusing harmful requests | Fading | Grok 4 0% | 1 | 8 | Aug 2025 | Sep 2026 | |
| Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. | OpenAI | Deception and alignment | Expert | In use | GPT-6.1 Terra 8.47% | 1 | 8 | Oct 2026 | Nov 2026 |
| Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. | OpenAI | Capture-the-flag challenges | Expert | Fading | GPT-5.6 Sol 97.1% | 1 | 8 | Apr 2026 | Aug 2026 |
| AgentHarm 110 malicious agent tasks with synthetic tools. Measures refusal and harmful task completion, plus benign controls. | Gray Swan AI + UK AI Security Institute | Refusing harmful requests | Fading | Grok 4 Fast 8.00% | 1 | 7 | Oct 2024 | Jun 2026 | |
| MT-AIME2024 Models solve the 30 AIME 2024 problems translated into 55 languages. A rule-based verifier checks each final answer. | Yonsei University, OneLineAI, and KAIST AI | Reasoning in many languages | Competition | Dropped | | 1 | 7 | Feb 2025 | May 2025 |
| QuALITY Models answer multiple-choice questions about English passages that average about 5,000 tokens. Scoring is accuracy. | New York University | Reasoning over long documents | School | Dropped | Claude 3 Opus 90.5% | 1 | 7 | Dec 2021 | Nov 2024 |
| WMDP Chemistry Models answer 408 multiple-choice questions that proxy hazardous chemical security knowledge. Chemistry split of WMDP. | CAIS + Scale AI et al. | Hazardous knowledge | Expert | Fading | Grok 4.20 90% | 1 | 7 | Mar 2024 | Oct 2026 |
| LV-Eval Models answer single-hop and multi-hop questions over contexts at five length levels from 16K to 256K tokens, with confusing facts inserted. | Infinigence AI and Tsinghua University (Yuan et al.) | Reasoning over long documents | Everyday | Dropped | | 1 | 7 | Feb 2024 | Jan 2025 |
| LongBench-Chat Models follow real user instructions over contexts of 10K to 100K tokens. GPT-4 scores each answer from 1 to 10 against references. | Tsinghua University and Zhipu AI (Bai et al., LongAlign) | Reasoning over long documents | Everyday | Dropped | | 1 | 6 | Jan 2024 | Jan 2025 |
| Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. | Google DeepMind | Video understanding | Everyday | Dropped | | 1 | 6 | May 2023 | Jul 2025 |
| Nexus Models write function calls for queries over real-world APIs, including nested and parallel calls. Scoring is the function calling success rate. | Nexusflow | Function calling | University | Dropped | | 1 | 6 | Dec 2023 | Nov 2024 |
| Qasper Models answer 5,049 questions about 1,585 NLP research papers from the full paper text. Answers are scored with token F1. | Allen Institute for AI (AI2) | Reasoning over long documents | University | Dropped | | 1 | 6 | May 2021 | Mar 2025 |
| SQuALITY Models write question-focused summaries of public-domain short stories. Outputs are scored against human-written reference summaries with ROUGE. | New York University | Reasoning over long documents | School | Dropped | | 1 | 6 | May 2022 | Mar 2025 |
| GSM-Plus Models solve GSM8K problems after perturbations such as changed numbers, added distractors and reversed questions. | Li et al. | Word problems | School | Dropped | GPT-4 85.6% | 1 | 5 | Feb 2024 | Nov 2025 |
| C3 Models answer 19,577 multiple-choice questions on Chinese dialogues and texts from Chinese-as-a-second-language exams. Scoring is accuracy. | Tencent AI Lab and Cornell University | Reading in many languages | School | Dropped | | 1 | 5 | Apr 2019 | Feb 2026 |
| BLEnD Models answer 52.6k questions about everyday culture in 16 countries and regions across 13 languages. Scoring is accuracy on short-answer and multiple-choice formats. | KAIST | Knowledge in many languages | Everyday | Dropped | | 1 | 5 | Jun 2024 | Sep 2025 |
| HarmBench Models answer harmful behavior requests, including copyright and contextual cases, with and without automated red-team attacks. A classifier judges whether the response complies. | Mazeika et al. (Center for AI Safety) | Jailbreaks and prompt injection | Everyday | Fading | | 1 | 5 | Feb 2024 | May 2026 |
| Refusals + System Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the system message. The metric is the answer rate. | xAI (SpaceXAI) | Jailbreaks and prompt injection | Fading | grok-code-fast-1 0% | 1 | 5 | Aug 2025 | Apr 2026 | |
| Refusals + User Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the user message. The metric is the answer rate. | xAI (SpaceXAI) | Jailbreaks and prompt injection | Fading | Grok 4 0% | 1 | 5 | Aug 2025 | Apr 2026 | |
| FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. | Irregular | Finding and exploiting real bugs | Research frontier | In use | GPT-6 Astra 38.1% | 1 | 5 | Jun 2026 | Nov 2026 |
| MedQA Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy. | MIT | Medicine | Expert | Dropped | GPT-4o 86.3% | 1 | 4 | Sep 2020 | Mar 2025 |
| RepoQA Models find a function in a long code repository from a description of what it does, across Python, C++, Rust, Java and TypeScript. | UIUC (EvalPlus team) | Finding facts in long inputs | University | Dropped | Claude 3 Opus 90.6% | 1 | 4 | Jun 2024 | Feb 2026 |
| TurkishMMLU Models answer over 10,000 multiple choice questions that Turkish curriculum experts wrote for nine high-school subjects. Scoring is accuracy. | Arda Yüksel et al. | Knowledge in many languages | School | Dropped | Claude 3 Opus 79.1% | 1 | 4 | Jul 2024 | Dec 2024 |
| IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. | MBZUAI | Knowledge in many languages | School | Dropped | | 1 | 4 | Oct 2023 | Dec 2024 |
| HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. | Center for AI Safety + Scale AI | Hallucination rates | Expert | Fading | | 1 | 4 | Jan 2025 | Oct 2026 |
| RoboSpatial-Home Models answer 350 spatial questions on home scenes for robots, in three types: object context, compatibility of placements and spatial configuration. | NVIDIA and The Ohio State University | Spatial reasoning and counting | Everyday | Dropped | | 1 | 4 | Nov 2024 | May 2026 |
| AMMLU Models answer MMLU multiple-choice questions across 57 subjects, translated into Arabic with GPT-4. Scored by accuracy. | FreedomIntelligence (AceGPT team, CUHK-Shenzhen) | Knowledge in many languages | University | Dropped | | 1 | 4 | Sep 2023 | Dec 2024 |
| CRPE Models answer single-choice questions about objects and their relations in images, covering subject, predicate and object. Each question is asked again with options rotated (CircularEval). | OpenGVLab, Shanghai AI Laboratory (All-Seeing Project V2) | General visual questions | Everyday | Dropped | | 1 | 4 | Feb 2024 | Nov 2025 |
| JMMLU Models answer 7,536 four-way multiple-choice questions in Japanese, from translated MMLU subjects and new Japan-specific subjects. Scored by accuracy. | Waseda University (nlp-waseda) | Knowledge in many languages | University | Dropped | | 1 | 4 | Jan 2024 | Dec 2024 |
| Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. | Qwen | Chat and instructions in many languages | Everyday | Dropped | | 1 | 4 | Sep 2024 | Dec 2024 |
| SycophancyEval (answer sycophancy) Models answer factual questions where the user suggests a wrong answer. Score is the relative accuracy drop against neutral prompts. | Anthropic (Sharma et al.) | Deception and alignment | Fading | Grok 4.20 4.00% | 1 | 4 | Oct 2023 | Apr 2026 | |
| WANDR 500 wide-search tasks: find tens to hundreds of entities and facts, then return structured records with citations. | Perplexity | Deep research | Everyday | In use | Claude Fable 5.2 82.1% | 1 | 4 | Jul 2026 | Nov 2026 |
| xAI sycophancy eval Factual questions paired with misleading user context. Rate at which the model abandons the correct answer. | xAI (SpaceXAI) | Wellbeing and sycophancy | Fading | Grok 4.5 0.01% | 1 | 4 | Sep 2026 | ||
| XQuAD Models answer 1,190 SQuAD questions over 240 paragraphs, professionally translated so the set is parallel across 11 languages. Scored by F1 and exact match. | DeepMind | Reading in many languages | Everyday | Dropped | | 1 | 4 | Oct 2019 | Mar 2025 |
| WildJailbreak Models face adversarial jailbreak prompts built from in-the-wild tactics, plus benign prompts that look harmful. Evaluations score refusals of harmful requests and compliance with benign ones. | Allen Institute for AI and University of Washington (WildTeaming) | Jailbreaks and prompt injection | Everyday | Fading | | 1 | 4 | Jun 2024 | Jan 2026 |
| EEBench 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge. | atopile | Electronics | Expert | In use | | 1 | 3 | Jul 2026 | Nov 2026 |
| BixBench3 Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts. | Edison Scientific | Research science | Research frontier | Not in cards | | Aug 2026 | Sep 2026 | ||
| BixBench v1 Replaced in Aug 2026 | FutureHouse | Lab and data work | Expert | Fading | Grok 4.5 93.8% | 1 | 3 | Feb 2025 | Sep 2026 |
| Point-Bench Models point to the image location a language request describes, in about 1,000 tasks across five reasoning categories. A point inside the target mask counts as correct. | Cheng et al. (PointArena team) | Detection and captioning | Everyday | Fading | | 1 | 3 | May 2025 | Mar 2026 |
| HELMET Models handle long-context tasks in seven groups including recall, RAG, re-ranking, many-shot learning, long QA and summarization, at set input lengths. | Princeton NLP (Yen et al.) | Reasoning over long documents | University | Dropped | GPT-4o mini 61.4% | 1 | 3 | Oct 2024 | Jul 2025 |
| MindCube Models answer 21,154 multiple choice questions about scenes seen from a few views. Questions test positions, orientations, and what-if movements. | MLL Lab, Northwestern University, with Stanford, NYU, and University of Washington | Spatial reasoning and counting | Everyday | Fading | | 1 | 3 | Jun 2025 | Oct 2026 |
| EMMA Models solve math, physics, chemistry and coding problems that need reasoning over text and images together, not each alone. Answers are checked against a key. | Hao et al. (EMMA team) | Visual math and diagrams | University | Fading | | 1 | 3 | Jan 2025 | Jun 2026 |
| Open ASR Leaderboard Speech recognition models transcribe the test sets of several English and multilingual corpora. Scored by average word error rate, with speed reported next to it. | Hugging Face (hf-audio) and collaborators | Speech recognition and translation | Everyday | Fading | | 1 | 3 | Sep 2023 | Oct 2026 |
| PubMedQA Models answer biomedical research questions with yes, no, or maybe, given a PubMed abstract without its conclusion. Scored by accuracy. | University of Pittsburgh and Carnegie Mellon University | Medicine | University | Dropped | Claude 3 Sonnet 78.3% | 1 | 3 | Sep 2019 | Apr 2024 |
| InfiniteBench/En.MC Models answer 229 multiple choice questions about long fake books with altered entity names, averaging about 184K tokens. Scoring is accuracy. | OpenBMB (Zhang et al.) | Reasoning over long documents | Everyday | Dropped | | 1 | 3 | Nov 2023 | Nov 2024 |
| MVPBench Models answer video questions on spatial, temporal and intuitive physics. Each question comes in a pair of near-identical videos with opposite answers, so text or single-frame shortcuts fail. | Meta FAIR | Physical prediction | Everyday | Dropped | | 1 | 3 | Jun 2025 | May 2026 |
| StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. | Meta | Jailbreaks and prompt injection | Expert | In use | | 1 | 3 | Oct 2026 | |
| WorldSense Models answer synthetic questions that test consistent reasoning about simple arrangements of entities described in text. | Meta FAIR (Benchekroun et al.) | Language reasoning | Everyday | Fading | | 1 | 3 | Nov 2023 | Apr 2026 |
| ZeroSCROLLS/QuALITY Models answer QuALITY multiple-choice questions about long stories zero-shot, as one task of the ZeroSCROLLS suite. Scoring is accuracy. | Tel Aviv University | Reasoning over long documents | Everyday | Dropped | | 1 | 3 | May 2023 | Nov 2024 |
| APPS Models write Python programs for 10,000 coding problems that range from introductory to competition level. Generated code runs against test cases. | UC Berkeley | Competitive programming | University | Dropped | Claude 3 Opus 70.2% | 1 | 3 | May 2021 | Mar 2024 |
| COCO Captions Models write a caption for each image, and outputs are compared with five human captions per image using BLEU, METEOR, ROUGE, and CIDEr. | Microsoft COCO | Detection and captioning | Everyday | Dropped | | 1 | 3 | Apr 2015 | Mar 2025 |
| HuggingFace Agent Benchmark Agents write Python code that calls Hugging Face Transformers tools to solve short tasks. Scores cover tool selection, tools used, and code correctness. | Hugging Face | Function calling | School | Dropped | | 1 | 3 | May 2023 | Sep 2023 |
| OK-VQA Models answer over 14,000 visual questions that need outside knowledge beyond the image. Scoring is VQA accuracy against human answers. | Allen Institute for AI and Carnegie Mellon University | General visual questions | Everyday | Dropped | | 1 | 3 | May 2019 | Mar 2025 |
| SpatialSense VQA Models judge whether a stated spatial relation between two objects holds in an image. The relations were crowdsourced to defeat simple cues. | Princeton Vision and Learning Lab (Kaiyu Yang, Olga Russakovsky, Jia Deng) | Spatial reasoning and counting | Everyday | Dropped | | 1 | 3 | Aug 2019 | Mar 2025 |
| MATH Level 5 Models solve the 1,324 hardest MATH test problems, drawn from competitions such as AMC and AIME. Scored by final answer match. | Dan Hendrycks et al. (UC Berkeley), subset run by Epoch AI | Competition problems with answers | Competition | Dropped | GPT-5 98.1% | 1 | 2 | Mar 2021 | Nov 2025 |
| RewardBench 2 Reward models must score one correct answer above three wrong ones on 1,865 new human prompts. Subsets cover factuality, precise instruction following, math, safety, focus and ties, and the score averages them. | Allen Institute for AI (Ai2) | Overall chat quality | University | Not in cards | | Jun 2025 | Jun 2025 | ||
| RewardBench v1 Replaced in Jun 2025 | Allen Institute for AI (Lambert et al.) | Everyday | Dropped | | 1 | 2 | Mar 2024 | Aug 2025 | |
| AIR-Bench 2024 Models respond to 5,694 harmful prompts across 314 risk categories drawn from 8 government regulations and 16 company policies. Category-specific LLM judges score refusals. | Stanford CRFM + Virtue AI et al. | Refusing harmful requests | Everyday | Dropped | Claude Haiku 4.5 93.2% | 1 | 2 | Jul 2024 | Jun 2026 |
| APEX-SWE Software-engineering productivity tasks spanning integration work and observability, such as diagnosing failures from telemetry. | Mercor | Fixing real repositories | Expert | In use | Claude Opus 5.5 67.6% | 1 | 2 | Jan 2026 | Nov 2026 |
| GeoBench Models geolocate GeoGuessr photos from community maps by giving coordinates and a country. Scored by country accuracy and a distance-based score. | ccmdi | General visual questions | Everyday | Dropped | Claude Fable 5.2 95% | 1 | 2 | Mar 2025 | Nov 2026 |
| SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. | Columbia DAPLab + Vals AI | Capture-the-flag challenges | Expert | In use | GPT-6 Astra 56.9% | 1 | 2 | Aug 2026 | Nov 2026 |
| VPCT Models view an image of a ball, ramps, and three buckets and predict which bucket the ball falls into. Scored by accuracy on 100 problems. | Chase Brower | Physical prediction | Everyday | Fading | | 1 | 2 | Jan 2025 | Mar 2026 |
| MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. | Shanghai AI Laboratory (OpenCompass) | Long videos | Everyday | Dropped | GPT-4o 2.15 | 1 | 2 | Jun 2024 | Aug 2025 |
| MIA-Bench Models answer 400 image prompts that stack several strict instructions on format, length and content. An LLM judge scores how well each answer follows every instruction. | Apple | General visual questions | Everyday | Dropped | GPT-4o 91% | 1 | 2 | Jul 2024 | Jan 2026 |
| PixMo-Count Models count objects in natural images and give the exact number. Scored by accuracy on the test split. | Allen Institute for AI (Molmo and PixMo) | Detection and captioning | Everyday | Fading | | 1 | 2 | Sep 2024 | Jun 2026 |
| ScienceQA Models answer about 21,000 multiple-choice science questions from school curricula, many with images. Scored by accuracy. | UCLA and Allen Institute for AI | General visual questions | School | Dropped | | 1 | 2 | Sep 2022 | Mar 2025 |
| SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. | Shanghai Jiao Tong University, Shanghai AI Laboratory and others (InternSVG team) | Web and app front ends | Everyday | Fading | Claude Sonnet 4 0.92 | 1 | 2 | Aug 2025 | Mar 2026 |
| MMT-Bench Vision-language models answer 31,325 multiple-choice questions across many multimodal tasks such as recognition, localization, reasoning and planning. | Shanghai AI Laboratory | General visual questions | Expert | In use | Qwen-VL-Max 62.4% | 1 | 2 | Apr 2024 | Oct 2026 |
| CrossVid Models reason across several videos at once in 10 tasks, with 9,015 questions on 5,331 videos in single-choice, multiple-choice and open-ended formats. | Li et al. (CrossVid team) | Video understanding | Everyday | Fading | GPT-4.1 45.2% | 1 | 2 | Nov 2025 | Jun 2026 |
| JudgeBench Models act as judges and pick the correct response from hard pairs on knowledge, reasoning, math and coding. The label comes from objective correctness, not human preference. | UC Berkeley | Overall chat quality | Expert | Dropped | o3-mini 80.9% | 1 | 2 | Oct 2024 | May 2025 |
| MME-CC Models answer 1,173 expert-annotated vision-grounded questions in 11 task types grouped as spatial, geometric and visual knowledge reasoning. | ByteDance Seed and collaborators (Zhang et al.) | Spatial reasoning and counting | Everyday | Fading | GPT-5 40.3% | 1 | 2 | Nov 2025 | Mar 2026 |
| WildVision Vision-language models answer 500 real user image prompts from the WildVision Arena. A GPT-4 judge compares each answer with a fixed baseline model. | Allen Institute for AI and UCSB | General visual questions | Everyday | Dropped | Aria 74.1 | 1 | 2 | Jun 2024 | Nov 2025 |
| IHEval Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction. | Zhang et al. (Notre Dame + Amazon) | Jailbreaks and prompt injection | Everyday | In use | GPT-4o 70% | 1 | 2 | Feb 2025 | Oct 2026 |
| Mantis-Eval Models answer questions that need reasoning across several interleaved images. Scored by accuracy. | Dongfu Jiang, Xuan He et al. (TIGER-Lab, University of Waterloo) | General visual questions | Everyday | Dropped | | 1 | 2 | May 2024 | Oct 2025 |
| Parametric CAD Bench v3 Agents create and edit parametric FreeCAD models from text and engineering drawings. Programmatic checks award continuous task rewards for geometry and specification consistency. | gNucleus AI | CAD models | Expert | Not in cards | Claude Opus 5.5 61.0 | Sep 2026 | Sep 2026 | ||
| Parametric CAD Bench V1 Replaced in Sep 2026 | In use | 1 | 2 | May 2026 | Aug 2026 | ||||
| ViewSpatial-Bench Models answer 5,712 questions that locate objects from the camera view and from the view of people in the scene. Five task types test egocentric and allocentric reasoning. | Dingming Li, Yueting Zhuang et al. | Spatial reasoning and counting | Everyday | Fading | | 1 | 2 | May 2025 | Oct 2026 |
| CIMemories Models use persistent memory across tasks and must withhold attributes when context requires. Reports worst-case disclosure over five samples. | Meta (Mireshghallah et al.) | Refusing harmful requests | Everyday | In use | GPT-4o 14.8% | 1 | 2 | Nov 2025 | Oct 2026 |
| GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. | Fudan University (OpenLMLab) | Exam questions | School | Dropped | GPT-4 72.2% | 1 | 2 | May 2023 | Aug 2025 |
| MSEB Audio representations from any multimodal system are scored on eight core tasks, including transcription, classification, retrieval, reasoning, segmentation, clustering, reranking and reconstruction. | Google Research | Audio understanding and voice | Everyday | Fading | | 1 | 2 | Feb 2026 | Oct 2026 |
| NaturalCodeBench Models solve 402 Python and Java problems taken from real user queries to online coding services in 6 domains. Semi-automatically built tests check each solution. | Tsinghua University and Zhipu AI (THUDM) | Function-level problems | University | Dropped | GPT-4 Turbo (Jan 2024) 52.5% | 1 | 2 | May 2024 | Aug 2025 |
| WorldTravel Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data. | ByteDance Seed and collaborators (Wang et al.) | General assistants | Everyday | Fading | GPT-5.2 32.7% | 1 | 2 | Dec 2025 | Mar 2026 |
| HealthBench Hard Hallucinations Models respond to difficult health conversations that 2 or more physicians validated. The score is the hallucination rate, so lower is better. | OpenAI | Hallucination rates | Dropped | GPT-5 1.60% | 1 | 2 | Aug 2025 | Aug 2025 | |
| MM-ClawBench Agents complete common OpenClaw tasks such as study planning, office documents, scheduled research, and code maintenance. Scoring is accuracy. | MiniMax | General assistants | Everyday | Fading | Claude Opus 4.6 75.4% | 1 | 2 | Mar 2026 | May 2026 |
| WMDP Cyber Models answer multiple-choice questions on offensive cybersecurity knowledge. The cybersecurity split of the Weapons of Mass Destruction Proxy benchmark. | CAIS + Scale AI et al. | Finding and exploiting real bugs | Expert | Fading | Grok 4.20 91% | 1 | 2 | Mar 2024 | Apr 2026 |
| BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. | Gemini Team, Google | Documents, charts and text in images | School | Dropped | | 1 | 2 | Jun 2024 | Jul 2025 |
| ContPhy Models infer physical properties such as mass and density of fluids and soft bodies from videos and predict their dynamics. | Zheng et al. (ContPhy team) | Physical prediction | Everyday | Fading | | 1 | 2 | Feb 2024 | Jun 2026 |
| AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. | ByteDance Seed | Scientific code | Expert | Fading | | 1 | 2 | Dec 2025 | Mar 2026 |
| BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. | ByteDance Seed | Genomics and life sciences | Competition | Fading | | 1 | 2 | Dec 2025 | Mar 2026 |
| Complex Workflow (ByteDance Seed in-house) Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare. | ByteDance Seed | Following precise instructions | Everyday | Fading | | 1 | 2 | Dec 2025 | Mar 2026 |
| Customer Support Q&A (ByteDance Seed in-house) Models answer user questions from noisy enterprise knowledge base results, such as policy questions in e-commerce, government services and telecom. | ByteDance Seed | Customer service | Everyday | Fading | | 1 | 2 | Dec 2025 | Mar 2026 |
| Education (ByteDance Seed in-house) Models solve, grade, explain and write K to 12 problems across core subjects in teaching scenarios. | ByteDance Seed | Other professions | School | Fading | | 1 | 2 | Dec 2025 | Mar 2026 |
| Information Extraction (ByteDance Seed in-house) Models extract words, sentences or fields from meeting records, legal texts, contracts, product descriptions and knowledge bases. | ByteDance Seed | Documents and spreadsheets | Everyday | Fading | | 1 | 2 | Dec 2025 | Mar 2026 |
| Inter-GPS (Geometry3K) Models solve high school geometry problems that combine a diagram and text from the Geometry3K test set. Scored by accuracy. | UCLA and Sun Yat-sen University (Lu et al.) | Visual math and diagrams | School | Dropped | | 1 | 2 | May 2021 | Mar 2025 |
| Internal adversarially-designed factuality evaluations Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. | OpenAI | Hallucination rates | Everyday | In use | GPT-6 Luna 7.60% | 1 | 2 | Mar 2023 | Sep 2026 |
| NV Chat Benchmark Models answer chat prompts from an internal NVIDIA set. The Nemotron 3.5 Super VL card labels the score an LMArena v3 proxy. | NVIDIA | Overall chat quality | Everyday | In use | | 1 | 2 | Oct 2026 | Oct 2026 |
| Passkey Retrieval Models find a random passkey hidden at a random position in a long filler prompt. Scored by retrieval accuracy. | EPFL | Finding facts in long inputs | Everyday | Dropped | | 1 | 2 | May 2023 | Jan 2025 |
| RISE Agents answer real questions from human experts that need multi-step search and interaction with information-dense web pages. MiniMax has not published the scoring method. | MiniMax | Deep research | Expert | Fading | | 1 | 2 | Feb 2026 | May 2026 |
| MiniMax TTS Multilingual Test Set Text-to-speech models clone two Common Voice speakers per language and read 100 test sentences in each of 24 languages. Scored by word error rate and speaker similarity. | MiniMax | Speech recognition and translation | Everyday | In use | | 1 | 2 | May 2025 | Sep 2026 |
| Multilingual LibriSpeech Models transcribe read audiobook speech in eight languages from LibriVox. Scored by word error rate. | Meta AI (Pratap et al.) | Speech recognition and translation | Everyday | Fading | | 1 | 2 | Dec 2020 | Dec 2025 |
| Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. | Vals AI | Finance | Expert | In use | Claude Fable 5.2 77.4% | 1 | 1 | Oct 2026 | Nov 2026 |
| Excel Modeling Benchmark 1.0 Replaced in Oct 2026 | Not in cards | Claude Fable 5.1 76.7% | Jul 2026 | Oct 2026 | |||||
| CorpFin v2 Models answer expert-written questions about credit agreements that often exceed 200 pages. An LLM judge grades each answer for accuracy. | Vals AI | Finance | Expert | In use | Claude Opus 5 73.2% | 1 | 1 | Aug 2026 | |
| SAGE v1.1 (Vals) Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading. | Vals AI | Other | University | In use | Claude Opus 5.5 91.3% | 1 | 1 | Oct 2026 | Nov 2026 |
| SAGE (Vals) v1 Replaced in Oct 2026 | Not in cards | Claude Opus 4.7 56.1% | Oct 2025 | Oct 2026 | |||||
| Open Arabic LLM Leaderboard v2 Models are scored on native Arabic tasks (AlGhafa native subsets, ArabicMMLU, Exams, MadinahQA, AraTrust, ALRAGE, ArbMMLU-HT) and ranked by the average. | Open Arabic LLM Leaderboard team (2A2I and community) with Hugging Face | Knowledge in many languages | School | Dropped | | 1 | 1 | Feb 2025 | Nov 2025 |
| Open Arabic LLM Leaderboard v1 Replaced in Feb 2025 | Open Arabic LLM Leaderboard team (2A2I and TII) with Hugging Face | Not in cards | | May 2024 | Jan 2025 | ||||
| BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. | BVB authors (Tang et al.) | Video understanding | Expert | In use | Claude Fable 5.2 76.4 | 1 | 1 | Sep 2026 | Nov 2026 |
| KINA Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key. | 2077AI (Jin et al.) | Expert-written questions | Expert | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| PHYSICS-Verified Corrected PHYSICS test problems with answer-level accuracy. The complete verified test set. | Yale University (Kaiyue Feng, Arman Cohan et al.) | Science questions | Expert | Not in cards | GPT-6 Astra 91.1% | Sep 2026 | Sep 2026 | ||
| PHYSICS Original Replaced in Sep 2026 | Dropped | o3-mini 59.9% | 1 | 1 | Mar 2025 | Aug 2025 | |||
| ArabicMMLU Models answer 14,575 multiple-choice questions in Modern Standard Arabic across 40 tasks, taken from school exams in North Africa, the Levant and the Gulf. | MBZUAI and collaborators (Koto et al.) | Knowledge in many languages | School | Dropped | | 1 | 1 | Feb 2024 | Nov 2025 |
| AraTrust Models answer 522 human-written Arabic multiple-choice questions on truthfulness, ethics, safety, physical and mental health, unfairness, illegal activities, privacy and offensive language. | Alghamdi et al. | Deception and alignment | Everyday | Dropped | | 1 | 1 | Mar 2024 | Nov 2025 |
| AgentIF (2026-10 setting) Models follow the same AgentIF instructions in thinking and non-thinking modes with vendor decoding settings. A gpt-5.6-luna judge scores 3 runs per model. | Tsinghua University KEG (THU-KEG) | Following precise instructions | Everyday | Not in cards | | Oct 2026 | Oct 2026 | ||
| AgentIF Original Replaced in Oct 2026 | In use | o1-mini 59.8% | 1 | 1 | May 2025 | Oct 2026 | |||
| MixEval Models answer real user queries matched to similar questions from existing benchmarks. Ground-truth answers make it a cheap proxy for Chatbot Arena rankings. | National University of Singapore (Ni et al.) | Overall chat quality | Everyday | Dropped | Claude 3.5 Sonnet 89.9% | 1 | 1 | Jun 2024 | Jul 2025 |
| SpaCE-10 Models answer more than 5,000 questions about indoor 3D scenes that test 10 atomic and 8 compositional spatial abilities. | SpaCE-10 team (Ziyang Gong et al.) | Spatial reasoning and counting | Everyday | Dropped | | 1 | 1 | Jun 2025 | Sep 2025 |
| KMMLU-Redux Models answer multiple-choice questions from Korean National Technical Qualification exams. The set is rebuilt from KMMLU with erroneous questions removed. | LG AI Research and collaborators | Exam questions | University | Dropped | o1 81.1% | 1 | 1 | Jul 2025 | Jul 2025 |
| MCP-Universe Agents complete tasks through 11 real MCP servers across six domains, such as navigation, repositories, finance, and browsing. Scored by execution-based success rate. | Salesforce AI Research | Apps and MCP servers | University | Fading | | 1 | 1 | Aug 2025 | Dec 2025 |
| C3-Bench Agents complete multi-task tool use dialogues built to stress tool dependencies, long context dependencies and frequent switches between policy types. | Tencent Hunyuan | Function calling | University | Dropped | Claude Sonnet 4 12.5% | 1 | 1 | May 2025 | Aug 2025 |
| Encyclo-K Models judge questions composed at test time from knowledge statements extracted from authoritative textbooks. Random composition is meant to resist contamination and test several facts at once. | ByteDance Seed and collaborators (Liang et al.) | Expert-written questions | University | Fading | GPT-5.1 62.1% | 1 | 1 | Dec 2025 | Feb 2026 |
| StructEval Models generate or convert structured outputs in 18 formats, such as JSON, TOML and HTML. StructEval-T covers text-only formats and StructEval-V renderable ones. | TIGER-AI-Lab (Yang, Jiang et al.) | Data science and SQL | Everyday | In use | | 1 | 1 | May 2025 | Oct 2026 |
| ChemBench Models answer more than 2,700 chemistry questions on knowledge and reasoning. Results are compared with a group of human chemists. | Lamalab, Friedrich Schiller University Jena | Science questions | Expert | Dropped | Claude Opus 4.7 70.9% | 1 | 1 | Apr 2024 | May 2026 |
| OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. | Shanghai Jiao Tong University GAIR Lab | Science olympiads | Competition | Dropped | GPT-4o 40.5% | 1 | 1 | Jun 2024 | Jun 2025 |
| HumanEval-V Models read a diagram and a function signature, then write Python code that must pass test cases. Tasks span six types of visual reasoning. | Fengji Zhang, Jacky Keung et al. | Function-level problems | University | Fading | o1 40.6% | 1 | 1 | Oct 2024 | Jan 2026 |
| MHPP Models write Python functions for 210 human-curated problems that need careful reading of specifications and multi-step reasoning. Unit tests check each solution. | SparksofAGI (Dai et al.) | Function-level problems | University | Dropped | o1-preview 68.1% | 1 | 1 | May 2024 | Jun 2025 |
| SGP-Bench Models read symbolic graphics programs such as SVG and CAD code and answer questions about what the rendered image would show, without seeing the image. | Max Planck Institute for Intelligent Systems and others (Zeju Qiu, Weiyang Liu et al.) | General visual questions | University | Dropped | Claude 3.5 Sonnet 67.4% | 1 | 1 | Aug 2024 | Aug 2025 |
| Math Kangaroo 2025 (MathArena) Vision language models answer 168 multiple-choice problems from the 2025 Math Kangaroo, given as screenshots across six grade groups. Many problems need reading a diagram. | MathArena (ETH Zurich SRI Lab and INSAIT) | Visual math and diagrams | School | Fading | GPT-5.5 94.9% | 1 | 1 | Oct 2025 | Jul 2026 |
| Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. | Center for AI Safety and Scale AI | Tasks from real jobs | Expert | In use | Claude Fable 5.2 41.3% | 1 | 1 | Oct 2025 | Nov 2026 |
| SWE-EVO Coding agents implement release-note changes in seven mature Python projects, editing about 21 files per task. Large test suites check each of the 48 tasks. | Le, Thai et al. | Fixing real repositories | Expert | Fading | GLM-4.7 39.6% | 1 | 1 | Dec 2025 | Mar 2026 |
| KRETA Models read and reason over Korean text in images across diverse visual contexts. It is a text-rich visual question answering test for Korean. | KRETA authors (tabtoyou et al.) | Documents, charts and text in images | Everyday | Fading | | 1 | 1 | Aug 2025 | Apr 2026 |
| MathCanvas-Bench Models solve 3,000 math problems that call for interleaved visual and textual solutions, such as drawing auxiliary lines in geometry. | Shi et al. (MathCanvas team) | Visual math and diagrams | School | Fading | | 1 | 1 | Oct 2025 | Feb 2026 |
| MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. | Carnegie Mellon University (CUBE lab) and collaborators | Video understanding | Everyday | Fading | Claude 3.5 Sonnet 64.1% | 1 | 1 | Mar 2025 | Dec 2025 |
| MME-VideoOCR Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning. | MME team (Shi et al., arXiv 2505.21333) | Video understanding | Everyday | Fading | | 1 | 1 | May 2025 | Apr 2026 |
| AgentWorldBench Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions. | Qwen Team, Alibaba Group | Interactive world models | University | In use | GPT-5.4 58.3 | 1 | 1 | Jun 2026 | Jun 2026 |
| KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. | HAE-RAE | General visual questions | University | Fading | | 1 | 1 | Apr 2026 | Apr 2026 |
| MORSE-500 Models answer questions embedded in 500 scripted video clips across six reasoning categories, including abstract, physical, planning, spatial and temporal reasoning. | Cai et al. (MORSE-500 team) | Video understanding | Everyday | Fading | o3 23.6% | 1 | 1 | Jun 2025 | Apr 2026 |
| Sudoku-Bench Models solve Sudoku variants with unusual rule sets that need creative, multi-step deduction. Scored by the share of puzzles solved. | Sakana AI | Abstract puzzles and logic | Competition | Dropped | o3-mini 14% | 1 | 1 | May 2025 | Sep 2025 |
| K-Viscuit Models answer multiple-choice visual questions about Korean culture. Questions were written through human and VLM collaboration. | Seoul National University and collaborators (ddehun et al.) | General visual questions | Everyday | Fading | GPT-4o 89.5% | 1 | 1 | Jun 2024 | Apr 2026 |
| VideoEval-Pro Video models give open-ended short answers to questions that need the whole long video, which removes the guessing that multiple choice allows. | TIGER-Lab and collaborators (Ma et al.) | Long videos | Everyday | Fading | GPT-4.1 40.8% | 1 | 1 | May 2025 | Apr 2026 |
| AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. | Cao et al. (arXiv 2605.27705) | Video generation | Expert | In use | GPT-5.6 Sol 38.4% | 1 | 1 | May 2026 | Sep 2026 |
| LongMemEval-V2 Memory systems help agents learn the affordances, dynamics and failure modes of customized web environments. Questions check whether that experience was internalized. | UCLA NLP | Learning from the context | Expert | Not in cards | | May 2026 | Feb 2026 | ||
| LongMemEval v1 Replaced in May 2026 | LongMemEval team (Wu et al., ICLR 2025) | Finding facts in long inputs | Everyday | In use | GPT-4o 92.4% | 1 | 1 | Oct 2024 | Oct 2026 |
| OIBench Models write programs for 250 original olympiad in informatics problems. Hidden tests check correctness, and time and space completion curves compare efficiency with reference solutions. | AGI-Eval (Meituan) | Competitive programming | Competition | Fading | o4-mini 36.4% | 1 | 1 | Jun 2025 | Jan 2026 |
| ProteinLMBench Models answer 944 hand-checked multiple choice questions on protein sequence, structure and function. | Johns Hopkins University, University of Cambridge, Shanghai AI Laboratory and others (Yiqing Shen et al.) | Genomics and life sciences | Expert | Dropped | GPT-4 Turbo 57.9% | 1 | 1 | Jun 2024 | Aug 2025 |
| TemporalBench Video models answer about 10K questions on fine-grained temporal dynamics, built from about 2K detailed human annotations of video clips. | Cai et al. (TemporalBench team) | Video understanding | Everyday | Dropped | LLaVA-OneVision 72B 28.1% | 1 | 1 | Oct 2024 | May 2025 |
| UniClawBench Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill. | The University of Hong Kong (HKU MMLab) | General assistants | Everyday | In use | Claude Opus 4.8 47.5% | 1 | 1 | Jul 2026 | Sep 2026 |
| When2Call Models decide when to call a tool, when to ask a follow-up question and when to say the tools cannot answer. Scored on multiple-choice and LLM-judged test sets. | NVIDIA | Function calling | Everyday | In use | GPT-4 Turbo 64.6% | 1 | 1 | Apr 2025 | Oct 2026 |
| LongAudioSpan Models answer 3,240 questions about 360 English and Chinese recordings from 10 minutes to over 2 hours. Each question is scored as multiple choice and as open-ended. | Huang et al. (arXiv 2608.26431, with Qwen authors) | Audio understanding and voice | Everyday | In use | GPT-5.4 80.2% | 1 | 1 | Aug 2026 | Sep 2026 |
| TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. | Nanjing University, ARC Lab (Tencent PCG) and Shanghai AI Lab | Video understanding | Everyday | Fading | | 1 | 1 | Dec 2025 | Sep 2026 |
| VCR-Bench Video models answer 1,034 questions on 859 videos, and graders score both the final answer and each tagged chain-of-thought step against human rationales. | Qi et al. (VCR-Bench team) | Video understanding | Everyday | Fading | o1 56.7% | 1 | 1 | Apr 2025 | Mar 2026 |
| DREAM-1K Video models write fine-grained descriptions of 1,000 videos from varied sources. An automatic method scores how many described events match the reference. | ByteDance Research (Tarsier team) | Video captioning | Everyday | Dropped | GPT-4V 34.4% | 1 | 1 | Jul 2024 | May 2025 |
| GaRAGe Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages. | Amazon (amazon-science) | Faithful to sources | Everyday | In use | | 1 | 1 | Jun 2025 | Oct 2026 |
| CSAT Math 2025 Models solve the mathematics sections of South Korea's 2025 College Scholastic Ability Test in Korean. The score averages the three elective subjects: calculus, statistics and geometry. | Korea Institute for Curriculum and Evaluation (exam), LG AI Research (evaluation) | Competition problems with answers | School | Dropped | | 1 | 1 | Nov 2024 | Mar 2025 |
| Multi-SWE-bench mini Agents fix real GitHub issues in 8 programming languages, a 400-instance subset of Multi-SWE-bench made for cheaper runs. Scored by tests passing. | ByteDance Seed | Fixing real repositories | Expert | Dropped | | 1 | 1 | Apr 2025 | Jul 2025 |
| ProcBench Models follow explicit step-by-step procedures to answer questions, which isolates multi-step inference from knowledge and search. | Fujisawa et al. | Abstract puzzles and logic | Everyday | Fading | o1-preview 69.8% | 1 | 1 | Oct 2024 | Feb 2026 |
| LVOmniBench Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track. | Tao et al. (arXiv 2603.19217) | Long videos | Everyday | In use | | 1 | 1 | Mar 2026 | Sep 2026 |
| ODV-Bench Streaming video models answer real-time questions about autonomous driving footage. | Zeng et al. (StreamForest team) | Video understanding | Everyday | Fading | | 1 | 1 | Sep 2025 | Apr 2026 |
| OmniCap-IF Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance. | Nanjing University (NJU-LINK Lab) and others | Video captioning | Everyday | In use | | 1 | 1 | Jun 2026 | Sep 2026 |
| SecCodeBench Models write or patch functions in project scaffolds drawn from real Alibaba vulnerabilities. Code must pass functional tests, then proof-of-concept security tests. | Alibaba Group (with Tsinghua, Zhejiang, Fudan, and Peking universities) | Secure code and review | University | Fading | | 1 | 1 | Jul 2025 | Jun 2026 |
| UNO-Bench Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model. | Meituan | Audio understanding and voice | Everyday | Dropped | | 1 | 1 | Oct 2025 | Dec 2025 |
| WebExploitBench Agents find and exploit 110 vulnerabilities in 15 real web applications, each in its own container. A verifier checks that the agent's proof of concept hits the same endpoint as the reference exploit. | AgentCyberRange | Finding and exploiting real bugs | Expert | In use | GPT-5.5 36.4% | 1 | 1 | Jun 2026 | Nov 2026 |
| DeR2 Bench Models reason over supplied scientific documents in a sandbox that separates evidence access from reasoning. Four regimes range from instructions only to relevant documents plus distractors. | ByteDance Seed and collaborators (Ying et al.) | Deep research | Expert | Fading | GPT-5.2 83.8% | 1 | 1 | Jan 2026 | Feb 2026 |
| Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. | Qwen Team, Alibaba Group | Chat and instructions in many languages | Everyday | Dropped | Claude 3 Opus 4.15 | 1 | 1 | Jun 2024 | Jul 2024 |
| StreamBench (StreamChat) Video models answer questions on streaming videos across media types, including multi-turn interaction and reasoning tasks. | Xiong et al. (StreamChat team) | Video understanding | Everyday | Dropped | GPT-4o mini 59.1% | 1 | 1 | Jan 2025 | May 2025 |
| VBench-2.0 Video generation models are scored on intrinsic faithfulness, such as human fidelity, controllability, creativity, physics and commonsense, beyond pixel quality and basic prompt match. | NTU S-Lab and Shanghai AI Laboratory (Vchitect) | Video generation | Everyday | Not in cards | | Mar 2025 | Sep 2025 | ||
| VBench 1.0 Replaced in Mar 2025 | Fading | | 1 | 1 | Nov 2023 | Feb 2026 | |||
| Agent Startup Bench Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews. | ByteDance Seed | Running a business | Expert | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| HealthBench-Hallu Models answer HealthBench-style medical prompts. A pipeline splits each answer into medical claims and checks them against medical evidence. The score is a weighted hallucination rate. | Baichuan AI | Hallucination rates | Expert | Fading | | 1 | 1 | Jan 2026 | Feb 2026 |
| MusTBench Models answer five kinds of music questions that need the correct time region of the audio, such as instrument entries. Music experts validated the items. | Kwon et al. (arXiv 2605.29300) | Audio understanding and voice | Expert | In use | | 1 | 1 | May 2026 | Sep 2026 |
| RelyToolBench Agents answer queries with simulated APIs built on the StableToolBench server. Scoring targets tool hallucinations, such as calling wrong or missing tools. | X-LANCE (Shanghai Jiao Tong University) | Function calling | Everyday | In use | | 1 | 1 | Dec 2024 | Oct 2026 |
| SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. | ByteDance Seed | General assistants | Everyday | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| AndroidDaily Agents complete 350 daily tasks in 94 real closed-source Android apps. A process-aware reviewer model checks each trajectory against written guidelines. | StepFun | Phones | Everyday | Fading | | 1 | 1 | Nov 2025 | May 2026 |
| CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. | Zhipu AI (Z.ai) | Fixing real repositories | Expert | Fading | GLM-5 25.8% | 1 | 1 | Feb 2026 | Apr 2026 |
| CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. | Zhipu AI (Z.ai) | Fixing real repositories | Expert | Fading | GLM-5V-Turbo 72.2% | 1 | 1 | Feb 2026 | Apr 2026 |
| ComplexNLU (Tencent) Models answer Tencent in-house natural language understanding tasks with complex inputs. Task and scoring details are not published. | Tencent Hunyuan | Reading comprehension | Everyday | Dropped | | 1 | 1 | Jun 2025 | Jun 2025 |
| CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. | ByteDance Seed | Desktop apps | Everyday | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| Doubao Multi-Turn Bench Models answer the turn of a real Doubao conversation where the user was dissatisfied, with the earlier turns as context. Rubrics grade each answer. | ByteDance Seed | Multi-turn conversation | Everyday | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| GEdit-Bench Image editing models follow real-world user edit instructions. A model judge scores semantic consistency, perceptual quality and an overall score. | StepFun (Step1X-Edit team) | Image generation and editing | Everyday | Dropped | | 1 | 1 | Apr 2025 | Nov 2025 |
| Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. | Google DeepMind | Embodied reasoning | Everyday | In use | | 1 | 1 | Jul 2026 | Jul 2026 |
| HumMusQA Models answer 320 hand-written questions about music, curated and checked by people with musical training. | Music Technology Group, Universitat Pompeu Fabra | Audio understanding and voice | Expert | In use | | 1 | 1 | Mar 2026 | Sep 2026 |
| Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. | ByteDance Seed | Spatial reasoning and counting | Everyday | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| MobileMiniWoB++ Agents complete MiniWoB++ web tasks inside the Android emulator of AndroidWorld. Scored by task success rate. | Google Research (AndroidWorld team) | Phones | Everyday | Dropped | | 1 | 1 | May 2024 | Feb 2025 |
| OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. | Meta FAIR | Embodied reasoning | Everyday | Dropped | | 1 | 1 | Apr 2024 | Sep 2025 |
| SaaS-Bench 106 long-horizon GUI workflows across 23 locally deployed real SaaS apps. Scored from app databases with weighted checkpoints and full-resolution rate. | UniPat AI + Peking University | Web browsing | University | In use | Claude Opus 5 31.1% | 1 | 1 | May 2026 | Aug 2026 |
| SpotSoundBench Models find the time span of short sound events hidden in long, unstructured YouTube recordings. It has 400 audio, query and timestamp triplets. | Sun et al. (SpotSound, arXiv 2604.13023) | Audio understanding and voice | Everyday | In use | | 1 | 1 | Apr 2026 | Sep 2026 |
| xDailyBench Models answer open-ended requests grounded in real user needs across more than 30 daily-life, study and office scenarios. Rubrics grade completion, correctness, coverage and usefulness. | ByteDance Seed | Overall chat quality | Everyday | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. | Zhipu AI (Z.ai) | Web and app front ends | Expert | Fading | GLM-5 74.8% | 1 | 1 | Feb 2026 | Apr 2026 |
| IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. | ByteDance Seed (BAGEL team) | Video generation | Everyday | Dropped | | 1 | 1 | May 2025 | May 2025 |
| ViDoRe V3 Retrieval and RAG pipelines answer queries over visually rich documents, such as reports with tables, charts and images. The benchmark has 10 datasets from professional domains, about 26,000 pages and 3,099 human-verified queries, each in 6 languages. It scores retrieval relevance, bounding-box grounding and the final answers. | Illuin Technology and NVIDIA | Documents, charts and text in images | Everyday | Not in cards | | Jan 2026 | Jan 2026 | ||
| ViDoRe v2 Replaced in Jan 2026 | Illuin Technology | University | Fading | | 1 | 1 | Jan 2025 | May 2026 | |
| AyaVisionBench Vision-language models answer image questions in 23 languages across 9 task categories, 135 image-question pairs per language. A model judge compares two models' answers and reports win rates. | Cohere For AI (Cohere Labs) | General visual questions | Everyday | Dropped | | 1 | 1 | Mar 2025 | May 2025 |
| CCPM Models pick which of four classical Chinese poetry lines matches a given modern Chinese translation. Scored by accuracy. | Tsinghua University (THUNLP) | Reading in many languages | School | Dropped | | 1 | 1 | Jun 2021 | Dec 2024 |
| Code2Video-AA Artificial Analysis' implementation of HeyGen's Code2Video Bench: models write a HyperFrames HTML composition for each of 114 product and brand video briefs. A model judge compares two rendered videos at a time on five axes, and the comparisons become Elo ratings. | Artificial Analysis | Web and app front ends | Expert | In use | | 1 | 1 | Oct 2026 | Nov 2026 |
| HoneyBench v0.2-alpha Measures reward hacking: how often an agent games its grader instead of doing the task. Lower is better. | Goodhart Labs | Deception and alignment | Research frontier | In use | | 1 | 1 | Nov 2026 | Nov 2026 |
| IFEval (Mistral internal version) Models follow prompts with verifiable format constraints. Mistral uses its own fixed version of IFEval. | Mistral AI | Following precise instructions | Everyday | Dropped | | 1 | 1 | Jun 2025 | Jun 2025 |
| Ko-Arena-Hard v2 (Upstage) Models answer hard user prompts in Korean, judged against a baseline in the style of Arena-Hard v2. Upstage built it in house. | Upstage | Chat and instructions in many languages | Everyday | Fading | | 1 | 1 | Dec 2025 | Mar 2026 |
| LeetCode Weekly Contest Models solve 126 LeetCode contest problems from July to November 2023. A solution counts only if it passes all of its 20+ test cases. | DeepSeek | Competitive programming | University | Dropped | | 1 | 1 | Nov 2023 | Jan 2024 |
| MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. | UC Berkeley | Exam questions | University | Dropped | | 1 | 1 | Sep 2020 | Jul 2025 |
| Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. | Google DeepMind | Finding and exploiting real bugs | Expert | In use | | 1 | 1 | Oct 2026 | |
| RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. | University of Washington and collaborators (Wang et al.) | Simulated robots | Everyday | Fading | | 1 | 1 | Jul 2025 | May 2026 |
| StrategyQA Models answer 2,780 yes or no questions that need implicit multi-step reasoning, such as whether Aristotle used a laptop. Scoring is accuracy. | Mor Geva et al. (Tel Aviv University and Allen Institute for AI) | Language reasoning | Everyday | Dropped | PaLM 2-L 90.4% | 1 | 1 | Jan 2021 | Apr 2025 |
| ARKitScenes Models locate objects in indoor RGB-D scenes captured on iPads and output 3D bounding boxes. Scored by 3D detection average precision. | Apple | Detection and captioning | Everyday | Dropped | | 1 | 1 | Nov 2021 | Nov 2025 |
| Audio Understanding Benchmark (Mistral) Models answer hard questions about 40 long-form audio recordings. Mistral built it in-house to test audio understanding beyond transcription. | Mistral AI | Audio understanding and voice | Expert | Dropped | | 1 | 1 | Jul 2025 | Jul 2025 |
| Brevity Benchmark NVIDIA evaluation of short, direct answers for voice and device assistants. The card gives no link or method. | NVIDIA | Overall chat quality | Everyday | In use | | 1 | 1 | Oct 2026 | Oct 2026 |
| CodeSimpleQA Models answer short bilingual fact questions about programming concepts and implementations across languages and computer science domains. | Yang et al. | Short factual questions | University | Fading | | 1 | 1 | Dec 2025 | Feb 2026 |
| CVTG-2K Image models render text in images across regions and languages. CVTG-2K reports normalized edit distance, CLIPScore and word accuracy. | Z.ai | Image generation and editing | Everyday | Fading | GLM-Image 0.91 | 1 | 1 | Jan 2026 | Jan 2026 |
| Docci Models retrieve matching documents and images using the DOCCI corpus. | Google DeepMind | Documents, charts and text in images | University | Fading | | 1 | 1 | May 2026 | |
| EIFBench Models follow extremely complex instructions that bundle several tasks and many constraints in one request, as in real product workflows. | Zou, Zhang et al. | Following precise instructions | Everyday | Fading | | 1 | 1 | Jun 2025 | Mar 2026 |
| Infographic factuality (Google) Image generation systems produce infographics and an automated rater scores factuality. | Google DeepMind | Hallucination rates | Everyday | In use | | 1 | 1 | Oct 2026 | Oct 2026 |
| Information Processing (ByteDance Seed in-house) Models analyze large collections of user-generated content, for example summarizing viewpoints and judging overall sentiment. | ByteDance Seed | Documents and spreadsheets | Everyday | Fading | | 1 | 1 | Dec 2025 | Mar 2026 |
| InsCtrl (Tencent) Models write text under detailed instructions in a Tencent in-house text creation test. Task and scoring details are not published. | Tencent Hunyuan | Following precise instructions | Everyday | Dropped | | 1 | 1 | Jun 2025 | Jun 2025 |
| InstructTTSEval Text-to-speech models follow natural-language style instructions in English and Chinese across acoustic parameter, descriptive style and role-play tasks. Gemini judges each clip. | Fudan University (Kexin Huang, Xipeng Qiu and others) | Speech recognition and translation | Everyday | Fading | | 1 | 1 | Jun 2025 | Jan 2026 |
| Intention Recognition (ByteDance Seed in-house) Models classify intent from dialogues, meeting transcripts, social posts and articles, such as purchase intent in customer service chats. | ByteDance Seed | Running a business | Everyday | Fading | | 1 | 1 | Dec 2025 | Mar 2026 |
| IPhO 2025 Models solve the theory problems of the 2025 International Physics Olympiad. Answers are graded against the official marking scheme. | International Physics Olympiad | Science olympiads | Competition | Fading | | 1 | 1 | Jul 2025 | Jun 2026 |
| LengthCtrl (Tencent) Models write text that must meet a requested length, in a Tencent in-house test. Task and scoring details are not published. | Tencent Hunyuan | Creative writing | Everyday | Dropped | | 1 | 1 | Jun 2025 | Jun 2025 |
| LongText-Bench Image models render long text in images. The benchmark reports separate English and Chinese scores. | Z.ai | Image generation and editing | Everyday | Fading | GLM-Image 0.97 | 1 | 1 | Jan 2026 | Jan 2026 |
| LVIS-MG Models ground many objects at once by outputting boxes for named categories in LVIS images. | ByteDance Seed | Detection and captioning | Everyday | Dropped | | 1 | 1 | May 2025 | May 2025 |
| m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. | Cohere | Customer service | Everyday | Dropped | | 1 | 1 | Apr 2025 | Aug 2025 |
| m-WildVision Vision-language models answer WildVision bench prompts machine-translated into 23 languages. A model judge compares two models' answers and reports win rates. | Cohere For AI (Cohere Labs) | General visual questions | Everyday | Dropped | | 1 | 1 | Mar 2025 | May 2025 |
| Mandarin-Minority translation (Tencent) Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation. | Tencent Hunyuan | Translation | Everyday | Dropped | | 1 | 1 | Sep 2025 | Sep 2025 |
| MeasureBench Vision-language models read values from images of measuring instruments, such as gauges and rulers, in real and synthesized photos. Scored by matching the reading to the key. | BAAI FlagEval team | General visual questions | Everyday | In use | | 1 | 1 | Oct 2025 | Jun 2026 |
| MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. | NVIDIA, Caltech and collaborators (Fan et al.) | Video games | Everyday | Fading | | 1 | 1 | Jun 2022 | Feb 2026 |
| MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. | Ai2 | Simulated robots | Everyday | Fading | | 1 | 1 | Feb 2026 | May 2026 |
| MSQA Models answer culture-specific knowledge questions in 11 major languages. | ByteDance Seed | Knowledge in many languages | University | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| MSR-VTT retrieval Models retrieve videos from text queries in the MSR-VTT collection. | Microsoft Research | Video understanding | University | Fading | | 1 | 1 | Jun 2016 | May 2026 |
| NExT-QA Models answer causal, temporal and descriptive questions about short videos of daily life. Labs usually report the multiple choice split. | National University of Singapore (Junbin Xiao et al.) | Video understanding | Everyday | Dropped | | 1 | 1 | May 2021 | Oct 2025 |
| OmniMMI Omni models answer 2,290 questions on 1,121 streaming videos, including proactive tasks where they must decide when to respond. | Wang et al. (OmniMMI team) | Video understanding | Everyday | Fading | | 1 | 1 | Mar 2025 | Mar 2026 |
| OneIG-Bench Image models generate images from prompts with English and Chinese evaluation splits. | Z.ai | Image generation and editing | Everyday | Fading | GLM-Image 0.53 | 1 | 1 | Jan 2026 | Jan 2026 |
| ProverBench (DeepSeek) Provers write Lean 4 proofs for 325 formalized problems: 15 from AIME 2024 and 2025 and 310 from textbooks across number theory, algebra, calculus and analysis. Lean checks each proof. | DeepSeek | Formal proofs | University | Dropped | | 1 | 1 | Apr 2025 | Apr 2025 |
| RAGTruth A word-level annotated corpus of hallucinations in retrieval-augmented generation answers. It is used to train and evaluate hallucination detection and RAG faithfulness. | ParticleMedia (NewsBreak) | Hallucination rates | Everyday | In use | | 1 | 1 | Jan 2024 | Oct 2026 |
| Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. | Meituan LongCat Team | Apps and MCP servers | University | Fading | | 1 | 1 | Jan 2026 | Jan 2026 |
| Robust IF Bench Models follow a diverse set of hard instructions from Microsoft AI's internal benchmark. | Microsoft AI | Following precise instructions | Everyday | In use | | 1 | 1 | Jun 2026 | Jun 2026 |
| ROK-FORTRESS Models face adversarial national security and public safety prompts in English and Korean, set in US or Korean contexts. Each prompt has a benign pair to measure over-refusal. | Scale AI | Dangerous capabilities | Expert | In use | | 1 | 1 | May 2026 | Aug 2026 |
| RoleBench Models answer prompts while playing one of many fixed characters from films and books. Answers are compared with reference role answers by automatic text overlap. | RoleLLM team (InteractiveNLP-Team, Wang et al.) | Overall chat quality | Everyday | In use | | 1 | 1 | Oct 2023 | Oct 2026 |
| RoleMRC Models play given roles in multi-turn chats, passage reading tasks and nested or prioritized instructions. The test split has 1.4k samples. | RoleMRC team (Lu et al., ACL 2025) | Following precise instructions | Everyday | In use | | 1 | 1 | Feb 2025 | Oct 2026 |
| ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. | Baichuan AI | Medicine | Expert | Fading | | 1 | 1 | Jan 2026 | Feb 2026 |
| TACoS Models find the time span in a cooking video that matches a sentence. Labs score temporal grounding by mean IoU with the annotated span. | Max Planck Institute for Informatics (Rohrbach et al.) | Video understanding | Everyday | Dropped | | 1 | 1 | Sep 2014 | May 2025 |
| tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Meituan LongCat Team | Customer service | Everyday | Fading | | 1 | 1 | Jan 2026 | Jan 2026 |
| TextCaps Models retrieve matching text and images using the TextCaps corpus. | Google DeepMind | General visual questions | University | Fading | | 1 | 1 | May 2026 | |
| TIIF-Bench Image models generate short and long images scored on overall image quality. | Z.ai | Image generation and editing | Everyday | Fading | GLM-Image 81.0 | 1 | 1 | Jan 2026 | Jan 2026 |
| U-Artifacts Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need. | ByteDance Seed | Web and app front ends | Everyday | Fading | | 1 | 1 | Dec 2025 | Mar 2026 |
| URO-Bench Spoken dialogue models answer speech with speech across Understanding, Reasoning and Oral conversation. It has a basic track of 16 datasets and a pro track of 20. | Yan et al. (arXiv 2502.17810) | Speech recognition and translation | Everyday | Fading | | 1 | 1 | Feb 2025 | Apr 2026 |
| VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Meituan LongCat Team | Apps and MCP servers | Everyday | Fading | | 1 | 1 | Jan 2026 | Jan 2026 |
| Word-Task (Tencent) Models solve Tencent in-house word level language tasks. Task and scoring details are not published. | Tencent Hunyuan | Reading comprehension | Everyday | Dropped | | 1 | 1 | Jun 2025 | Jun 2025 |
| YouCook2 retrieval Models retrieve cooking videos from text queries in the YouCook2 collection. | University of Illinois and collaborators | Video understanding | University | Fading | | 1 | 1 | Sep 2018 | May 2026 |
| BEAM Models answer probing questions about coherent multi-session user conversations of 128K, 500K, 1M and 10M tokens. Questions test ten memory abilities such as knowledge updates, temporal reasoning and contradiction resolution. | Mohammad Tavakoli, Hamed Zamani, J Ross Mitchell and collaborators | Learning from the context | Everyday | In use | 1 | 1 | Oct 2025 | Oct 2026 | |
| CCBench (MMBench Chinese Culture) Vision language models answer 510 Chinese-language multiple-choice questions about images of Chinese culture, such as calligraphy, painting, food and historical figures. | Shanghai AI Laboratory (OpenCompass, MMBench team) | General visual questions | Everyday | Dropped | | 1 | Oct 2023 | Jun 2025 | |
| FACTS Parametric V2 Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer. | Google DeepMind + Google Research + Kaggle | Short factual questions | University | Not in cards | GPT-6 Astra 83.8% | Oct 2026 | Oct 2026 | ||
| FACTS Parametric v1 Replaced in Oct 2026 | Fading | GPT-6 Astra 83.1% | 1 | Dec 2025 | Oct 2026 | ||||
| CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Vals AI | Finding and exploiting real bugs | Expert | In use | | 1 | Oct 2026 | Nov 2026 | |
| CyberBench (Vals) v1 Replaced in Oct 2026 | Not in cards | GPT-5.6 Sol 88.1% | Jun 2026 | Sep 2026 | |||||
| YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. | Collinear AI | Running a business | University | In use | Claude Fable 5.2 $3,445,495 | 1 | Feb 2026 | Nov 2026 | |
| EnterpriseBench: CoreCraft Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy. | Surge AI | Simulated companies | Expert | In use | Claude Fable 5.1 77.4% | 1 | Feb 2026 | Nov 2026 | |
| Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. | Google (Android Developers) | Building whole projects | Expert | Not in cards | Claude Sonnet 5.5 40% | Sep 2026 | Nov 2026 | ||
| AndroidBench 1.0 Replaced in Sep 2026 | Fixing real repositories | In use | Claude Opus 5 91.8% | 1 | Mar 2026 | Aug 2026 | |||
| DistressBench Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. | Scale AI | Wellbeing and sycophancy | Expert | In use | | 1 | Oct 2026 | Nov 2026 | |
| Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. | Vals AI | Composite work indices | Expert | In use | | 1 | Oct 2026 | Nov 2026 | |
| DAYJOB: Finance Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Tasks from real jobs | Expert | In use | Claude Opus 5.5 42.5% | 1 | Sep 2026 | Nov 2026 | |
| Humanity's Sixth Sense Models answer open-ended questions about what an image or short video implies but does not show, such as hidden objects, causes and intentions. An LLM judge checks each answer against a human-written rubric. | Scale AI and Elorian | General visual questions | Everyday | In use | | 1 | Oct 2026 | Nov 2026 | |
| Lakera B3 (Backbone Breaker Benchmark) Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. | Lakera + ETH Zurich + UK AISI + Oxford | Jailbreaks and prompt injection | In use | Claude Opus 4.5 89% | 1 | Oct 2025 | Oct 2026 | ||
| LiveSQLBench Contamination-resistant text-to-SQL on PostgreSQL with hierarchical knowledge bases, including CRUD tasks. Scored by soft execution match and test cases. | BIRD team (HKU) | Data science and SQL | University | In use | | 1 | May 2025 | Jun 2026 | |
| LAMBADA Models predict the last word of narrative passages where humans can guess it only from the whole passage. Scored by accuracy. | University of Trento and University of Amsterdam | Reading comprehension | Everyday | Dropped | PaLM 2-L 86.9% | 1 | Jun 2016 | Nov 2023 | |
| SWE Atlas Test Writing Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests. | Scale AI | Fixing real repositories | Expert | In use | Claude Fable 5.1 67% | 1 | Apr 2026 | Nov 2026 | |
| RefAdv-S Models locate the object a referring expression describes among hard distractors in real images. Scored by accuracy at 0.5 box overlap on 1,142 cases. | Northeastern University | Detection and captioning | Everyday | In use | | 1 | Feb 2026 | Aug 2026 | |
| MADQA Agents search 800 PDF documents to answer 2,250 human-written questions. Scoring covers answer accuracy and how much search effort the agent spends. | Łukasz Borchmann et al. (Snowflake, University of Oxford, and partners) | Documents, charts and text in images | Everyday | In use | | 1 | Mar 2026 | Aug 2026 | |
| sudo L7 Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change. | Surge AI | Fixing real repositories | Expert | In use | Claude Fable 5.2 52.2% | 1 | Oct 2026 | Nov 2026 | |
| HANDBOOK.md Agents do routine company work with mock email, chat, calendar, issue and commerce tools while following a 20 to 124 page handbook. Programmatic criteria check required and forbidden actions, and a trial passes only if all pass. | Surge AI | Following precise instructions | University | In use | Claude Opus 5.5 52.3% | 1 | Jul 2026 | Nov 2026 | |
| MolecularIQ Models answer questions about molecular graphs, such as counting, indexing and constrained generation, and a symbolic checker verifies each answer. | Johannes Kepler University Linz (Bartmann, Klambauer, Luukkonen et al.) | Genomics and life sciences | Expert | In use | gpt-oss-120b 47.5% | 1 | Jan 2026 | Sep 2026 | |
| SWE Atlas Refactoring Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result. | Scale AI | Fixing real repositories | Expert | In use | GPT-6 Astra 59% | 1 | May 2026 | Nov 2026 | |
| SWE-fficiency 498 tasks in nine Python data/ML/HPC repos: agent must speed up a real workload without breaking tests. Scored as speedup relative to expert patch. | SWE-fficiency authors (ICML 2026) | Speed and optimization | Expert | In use | Claude Opus 4.5 22.5% | 1 | Nov 2025 | Jun 2026 | |
| Alyah Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect. | Technology Innovation Institute (TII) and community contributors | Knowledge in many languages | Everyday | In use | | 1 | Jan 2026 | Oct 2026 | |
| PLawBench Model handles 850 realistic legal consultation, case-analysis and document-drafting tasks. LLM judge scores against about 12,500 expert rubric items. | SKYLENAGE-AI (Shi et al.) | Law | Expert | In use | GPT-5.2 69.7% | 1 | Jan 2026 | Aug 2026 | |
| SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. | Shanghai AI Lab and partners (Wanghan Xu et al.) | Research science | Expert | In use | | 1 | Dec 2025 | Sep 2026 | |
| OmniParseBench Document parsers convert single pages to text, and 16,288 binary unit tests on 2,937 pages check the output. The headline joins text, table and layout subscores. | Datalab | Documents, charts and text in images | Everyday | In use | | 1 | Oct 2026 | Oct 2026 | |
| RoadmapBench Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests. | UniPat AI | Building whole projects | Expert | In use | Claude Opus 4.7 39.1% | 1 | May 2026 | Sep 2026 | |
| Ego3D-Bench Models answer spatial reasoning questions about ego-centric multi-view scenes, such as those from self-driving cars. Some items ask for distances, scored by error. | Authors of arXiv 2509.06266 | Spatial reasoning and counting | Everyday | In use | | 1 | Sep 2025 | Jun 2026 | |
| GroundUI Agents get a screenshot and an instruction and must click the right UI element. About 18K test samples cover web, desktop and mobile screens. Scored by whether the point falls in the target box. | AgentStudio team (Longtao Zheng et al., Nanyang Technological University and Skywork AI) | Finding things on screen | Everyday | Dropped | | 1 | Jun 2024 | Mar 2025 | |
| SWE-bench-Live Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch. | Microsoft | Fixing real repositories | Expert | Dropped | | 1 | May 2025 | Sep 2026 | |
| Harbor-Index Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. | Harbor and Laude Institute (Terminal-Bench team) | Terminal tasks | Research frontier | In use | GPT-5.5 28.1% | 1 | Jun 2026 | Aug 2026 | |
| MarathonDocBench Models parse 100 documents spanning 103 to 300 pages. Evaluation scores content, cross-page continuations, figures, reading order, and heading hierarchy. | Baidu ERNIE Team | Documents, charts and text in images | Expert | In use | | 1 | Oct 2026 | Oct 2026 | |
| SWE Refactor Bench Agents migrate 20 whole open-source repositories to a new language, framework, platform, or build system. Graded by migration audit, behavioral tests, and adversarial verifier models. | Einsia | Building whole projects | Research frontier | In use | Claude Opus 5 47% | 1 | Aug 2026 | Oct 2026 | |
| SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix. | ETH Zurich and LogicStar.ai | Fixing real repositories | Expert | Fading | | 1 | Jun 2024 | Dec 2025 | |
| BrokenArXiv Models are asked to prove statements from recent arXiv papers that were changed to be plausible but false. Full credit goes to models that say the statement is false. | MathArena (ETH Zurich SRI Lab and INSAIT) | Research mathematics | Research frontier | In use | | 1 | Mar 2026 | Oct 2026 | |
| E-Bench-Code (Tencent) Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published. | Tencent Hy team | Documents and spreadsheets | Expert | In use | GPT-5.6 Sol 83.3% | 1 | Aug 2026 | Aug 2026 | |
| Finch (FinWorkBench) Agent completes 172 enterprise finance and accounting spreadsheet workflows (384 tasks) from real data. Human plus calibrated LLM-judge grading. | FinWorkBench (Dong et al.) | Finance | Expert | In use | Claude Opus 4.5 43% | 1 | Dec 2025 | Oct 2026 | |
| Functional MATH Models solve new instances of MATH problems that code generates for each snapshot. The gap from static MATH accuracy measures memorization. | Consequent AI | Competition problems with answers | Competition | Dropped | GPT-4 14.2% | 1 | Feb 2024 | May 2024 | |
| VLADBench Models answer closed-form questions about driving scenes, from static traffic knowledge to reasoning about dynamic road situations. | Authors of arXiv 2503.21505 | Spatial reasoning and counting | University | In use | | 1 | Mar 2025 | Jun 2026 | |
| Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. | Alibaba Qwen Team, Alibaba Cloud security and National University of Singapore | Secure code and review | Expert | In use | Claude Fable 5 25.1% | 1 | Oct 2026 | Nov 2026 | |
| MT-Bench-101 Models hold 1,388 multi-turn dialogues with 4,208 turns across 13 tasks from a three-tier ability taxonomy. GPT-4 scores each turn from 1 to 10. | Alibaba Group, CUHK MMLab and Shanghai AI Lab (Ge Bai, Jie Liu et al.) | Multi-turn conversation | Everyday | Dropped | | 1 | Feb 2024 | Jan 2025 | |
| PatchEval-Verified Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch. | ByteDance | Finding and exploiting real bugs | Expert | Not in cards | GPT-5.6 Sol 83.9% | Jul 2026 | Jul 2026 | ||
| PatchEval Original Replaced in Jul 2026 | In use | GPT-5.6 Sol 83.9% | 1 | Nov 2025 | Nov 2026 | ||||
| RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. | MMLab@HKU + UC Berkeley + Tsinghua + others (RoboDojo team) | Simulated robots | Everyday | In use | GPT-6 Astra 27.6% | 1 | Jul 2026 | Nov 2026 | |
| SITE Models answer multiple-choice spatial questions over single images, multiple images and video, drawn from about 30 vision datasets. Tasks span figural to environmental scale. | Boston University and Microsoft Research | Spatial reasoning and counting | Everyday | Fading | | 1 | May 2025 | Jan 2026 | |
| SWE-Perf Agents speed up code in 140 tasks drawn from performance pull requests in real GitHub repos. Scored on correctness and runtime gain. | TikTok and Xi'an Jiaotong University | Speed and optimization | Expert | Fading | Claude 3.7 Sonnet 2.26% | 1 | Jul 2025 | Dec 2025 | |
| VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. | QuantaAlpha | Video understanding | University | In use | | 1 | Jan 2026 | Aug 2026 | |
| GDPval-MM Agents produce office deliverables for GDPval tasks in a MiniMax agent framework. A judge model compares full trajectories pairwise and reports a win rate. | MiniMax | Tasks from real jobs | Expert | Fading | Claude Opus 4.6 73.5% | 1 | Feb 2026 | Feb 2026 | |
| Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score. | Moonshot AI | Building whole projects | Expert | In use | GPT-6 Astra 66.5% | 1 | Oct 2026 | Oct 2026 | |
| Kimi Hallu Bench Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. | Moonshot AI | Hallucination rates | Expert | In use | GPT-6 Astra 4.20% | 1 | Oct 2026 | Oct 2026 | |
| L-Eval Models answer over 2,000 queries on 508 long documents of 3k to 200k tokens across 20 subtasks. Scoring uses exact match and judge-model ratings. | OpenLMLab (Fudan University) | Reasoning over long documents | University | Dropped | Claude 1.3 76.4% | 1 | Jul 2023 | Feb 2024 | |
| Multi-SWE-bench flash Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing. | ByteDance Seed | Fixing real repositories | Expert | Dropped | | 1 | Jul 2025 | Jul 2025 | |
| OctoCodingbench Agents complete 72 coding tasks while following rules from system prompts, AGENTS.md files, skills, and memory. An LLM judge checks 2,422 pass or fail items. | MiniMax | Fixing real repositories | University | Fading | Claude Opus 4.5 36.2% | 1 | Dec 2025 | Dec 2025 | |
| Orak LLM agents play 12 popular video games across most genres through an MCP interface. Each game scores in-game progress or outcomes, and the board ranks models by their average rank across the games. | KRAFTON AI with Seoul National University, NVIDIA and UW-Madison | Video games | Everyday | Fading | | 1 | Jun 2025 | Mar 2026 | |
| QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. | Qwen Team, Alibaba Group | Web and app front ends | University | In use | | 1 | Aug 2026 | Aug 2026 | |
| QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. | Qwen (Alibaba) | Documents, charts and text in images | University | In use | | 1 | Aug 2026 | Aug 2026 | |
| TOMG-Bench Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. | Hong Kong Polytechnic University and Shanghai AI Lab (Jiatong Li et al.) | Genomics and life sciences | Expert | In use | GPT-3.5 Turbo 18.6% | 1 | Dec 2024 | Sep 2026 | |
| 3DSRBench Models answer questions about 3D spatial relations in images, such as height, location, orientation and relations between objects. | Wufei Ma, Alan Yuille et al. | Spatial reasoning and counting | Everyday | Fading | | 1 | Dec 2024 | Jan 2026 | |
| AVMeme Exam Models answer questions about over one thousand iconic internet sounds and videos, from surface content to context, emotion, usage and world knowledge. | Jiang et al. (AVMeme Exam team) | Audio understanding and voice | Everyday | Fading | | 1 | Jan 2026 | Apr 2026 | |
| miniCTX-v2 Provers prove Lean theorems from real projects using long repository context, with theorems taken after a November 28, 2024 cutoff. | CMU L3 Lab (Hu et al.) | Formal proofs | Expert | Dropped | | 1 | Mar 2025 | Sep 2025 | |
| Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. | Google DeepMind | Embodied reasoning | Everyday | In use | | 1 | Jul 2026 | Jul 2026 | |
| QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). | Alibaba Qwen | Fixing real repositories | Expert | In use | Claude Fable 5 63.1% | 1 | Aug 2026 | ||
| QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. | Alibaba Qwen | Web and app front ends | Expert | In use | Claude Fable 5 1,770 | 1 | Aug 2026 | ||
| QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo. | Alibaba Qwen | Web and app front ends | University | In use | GPT-5.6 Sol 1,758 | 1 | Aug 2026 | ||
| QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. | Qwen Team, Alibaba Group | Web and app front ends | University | In use | GPT-5.4 1,884 | 1 | May 2026 | Jun 2026 | |
| StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup. | StepFun | Fixing real repositories | Expert | In use | Claude Opus 5 63.9% | 1 | Sep 2026 | Sep 2026 | |
| SWE-Review Models review code changes in several languages and scenarios to find defects. Scores reflect defect recall and the rate of hallucinated defects. | MiniMax | Fixing real repositories | Expert | Fading | Claude Opus 4.5 16.2% | 1 | Dec 2025 | Dec 2025 | |
| ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. | Qwen Team, Alibaba Group | Function calling | Everyday | Fading | | 1 | Apr 2026 | Apr 2026 | |
| xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. | Moonshot AI | Finding and exploiting real bugs | Expert | In use | GPT-5.6 Sol 100% | 1 | Oct 2026 | Oct 2026 | |
| Z.ai Code Bench Zhipu in-house coding benchmark used for GLM-5.3. Reports pass rate and output tokens per task. Method unpublished. | Zhipu AI (Z.ai) | Fixing real repositories | Expert | In use | Claude Fable 5 39.5% | 1 | Aug 2026 | Aug 2026 | |
| Answer Factuality (Perplexity) Models answer questions from search results. Graders check that answers use facts grounded in the results and resolve conflicting or missing information. | Perplexity | Faithful to sources | Everyday | Dropped | | 1 | Feb 2025 | Feb 2025 | |
| CB Models label whether a text commits its speaker to an embedded clause as entailment, contradiction, or neutral. Scored by accuracy and F1. | Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser | Language reasoning | Everyday | Dropped | PaLM 2-L 87.5% | 1 | Apr 2019 | May 2023 | |
| FutureOmni Models watch and hear 919 videos and answer 1,034 multiple-choice questions that ask what will happen next. Questions need audio and visual cues together. | OpenMOSS (Fudan University) and collaborators | Video understanding | Everyday | Fading | MiMo-V2-Omni 66.7% | 1 | Jan 2026 | Mar 2026 | |
| Kimi Claw 24/7 Bench Agents work on 17 persistent multi-day professional scenarios through OpenClaw, with 610 checkpoints. The score is the mean pass rate over three runs. | Moonshot AI | Tasks from real jobs | Expert | In use | Kimi K2.7 Code 46.9% | 1 | Jun 2026 | Jun 2026 | |
| Mind2Web 2 Agents browse the live web for 130 long tasks and return answers with citations. Task-specific judge agents built from rubric trees grade correctness and attribution. | Ohio State University (OSU NLP Group) | Deep research | Expert | Not in cards | OpenAI deep research 28% | Jun 2025 | Apr 2025 | ||
| Mind2Web v1 Replaced in Jun 2025 | Web browsing | Everyday | Dropped | | 1 | Jun 2023 | Jul 2025 | ||
| MultiRC Models pick all correct answers to questions that need reasoning over several sentences of a paragraph. Scored by F1 over answer options. | University of Pennsylvania (CogComp) | Reading comprehension | School | Dropped | PaLM 2-L 88.2% | 1 | Jun 2018 | May 2023 | |
| WSC Models decide whether a pronoun refers to a given noun phrase in sentences that need commonsense to resolve. Scored by accuracy. | NYU, Facebook AI, University of Washington, and DeepMind (SuperGLUE) | Commonsense | Everyday | Dropped | PaLM 2-L 86.9% | 1 | May 2019 | May 2023 | |
| Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. | Google (Big Sleep team) | Finding and exploiting real bugs | Expert | In use | | 1 | Jul 2026 | Jul 2026 | |
| CHAOS-Bench Models read deliberately corrupted words in 500 academic page images. Scores measure faithful recognition of the visible strings instead of correction to plausible words. | Tencent Hunyuan | Documents, charts and text in images | Expert | In use | | 1 | Jul 2026 | Oct 2026 | |
| Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. | Google (Chrome Security) | Finding and exploiting real bugs | Expert | In use | | 1 | Jul 2026 | Jul 2026 | |
| RTE Models decide whether a premise sentence entails a hypothesis, using data from the PASCAL RTE challenges. Scored by accuracy. | PASCAL RTE Challenge organizers, packaged in GLUE | Language reasoning | Everyday | Dropped | PaLM 2-L 79.3% | 1 | Apr 2018 | Jul 2023 | |
| Android in the Zoo (AitZ) Models predict the next action on Android app screens, given the goal and past steps. Scored by exact match of the predicted action on the test split. | Zhang et al. (Fudan University and others) | Phones | Everyday | Dropped | | 1 | Mar 2024 | Jan 2025 | |
| Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. | Moonshot AI | Running a business | Expert | In use | Kimi K3.1 Pro 37.8% | 1 | Oct 2026 | Oct 2026 | |
| AdvancedMathBench Models write proofs for 245 advanced problems at undergraduate and doctoral qualifying exam levels. A verifier trained on expert annotations grades each proof. | Shanghai AI Lab (InternLM) | Written proofs | Expert | In use | | 1 | Jul 2026 | Sep 2026 | |
| AIR-Bench Foundation Models answer foundational audio understanding questions. The score is the official average accuracy across nineteen tasks. | Zhejiang University and Alibaba Group | Audio understanding and voice | Everyday | In use | | 1 | Feb 2024 | Oct 2026 | |
| Artificial Analysis average word error rate Speech transcription systems are scored by average word error rate across the Artificial Analysis evaluation set. | Artificial Analysis | Speech recognition and translation | Everyday | In use | | 1 | Aug 2026 | Aug 2026 | |
| Artificial Analysis Cyber Index Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. | Artificial Analysis and the Cyber Index Alliance | Finding and exploiting real bugs | Expert | In use | | 1 | Sep 2026 | Oct 2026 | |
| CELLO Models follow real-world complex instructions with multiple tasks, noisy inputs and format or length constraints. Rule-based checks score each answer against the instruction features. | Fudan University (He et al.) | Following precise instructions | Everyday | Dropped | | 1 | Sep 2023 | Mar 2025 | |
| ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. | Gemini Team, Google | Documents, charts and text in images | University | Dropped | | 1 | Jun 2024 | May 2024 | |
| ChineseQA Models answer Chinese trivia questions in a closed-book format modeled on TriviaQA. | DeepSeek | Trivia and open questions | Everyday | Dropped | | 1 | Nov 2023 | Nov 2023 | |
| CMRC Models extract answer spans from Chinese Wikipedia paragraphs for about 20,000 human-written questions. Scoring is exact match and F1. | Joint Laboratory of HIT and iFLYTEK Research (HFL) | Reading in many languages | Everyday | Dropped | | 1 | Oct 2018 | Dec 2024 | |
| COCO Models detect and locate objects in everyday photos from the Microsoft COCO dataset. Scored by mean average precision over box overlap thresholds. | Microsoft and academic partners | Detection and captioning | Everyday | In use | | 1 | May 2014 | Aug 2026 | |
| Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. | OpenAI | Fixing real repositories | Expert | Dropped | GPT-5-Codex 51.3% | 1 | Sep 2025 | Sep 2025 | |
| ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. | OpenAI | Function calling | Everyday | Dropped | gpt-realtime 66.5% | 1 | Aug 2025 | Aug 2025 | |
| Context Aware Automatic Speech Recognition Speech transcription systems are scored for semantic accuracy with and without free-form context. | OpenAI | Speech recognition and translation | Everyday | In use | GPT-Live-Transcribe 44.6% | 1 | Jul 2026 | Jul 2026 | |
| COPA Models choose which of two alternatives is the more plausible cause or effect of a premise, across 1,000 questions. Scored by accuracy. | University of Southern California (ICT) | Commonsense | Everyday | Dropped | PaLM 2-L 96% | 1 | Mar 2011 | May 2023 | |
| CueSpeak Models speak at the correct moment and give the expected semantic response during simultaneous speech. | Thinking Machines Lab | Speech recognition and translation | Everyday | Fading | | 1 | May 2026 | May 2026 | |
| Data Science Tasks (OpenAI) Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | OpenAI | Documents and spreadsheets | Expert | In use | GPT-6 Astra 40.9% | 1 | Sep 2026 | Sep 2026 | |
| Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | OpenAI | Fixing real repositories | Expert | In use | GPT-6 Astra 63.9% | 1 | Sep 2026 | Sep 2026 | |
| DECK-Arena Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores. | Moonshot AI | Documents and spreadsheets | University | In use | Kimi K3.1 Pro 1,456 | 1 | Oct 2026 | Oct 2026 | |
| Design Bench (Moonshot) Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. | Moonshot AI | Taste and style | University | In use | Kimi K3.1 Pro 957 | 1 | Oct 2026 | Oct 2026 | |
| Design Tasks (OpenAI) Models produce design work such as slides and documents that follow templates, from an OpenAI internal set. OpenAI has not published the grading. | OpenAI | Documents and spreadsheets | Expert | In use | GPT-6 Astra 50% | 1 | Sep 2026 | Sep 2026 | |
| DS-Arena-Code Models answer common coding requests in an internal subjective test. GPT-4o judges responses against competitor models to give a win rate. | DeepSeek-AI | Function-level problems | University | Dropped | | 1 | Sep 2024 | Sep 2024 | |
| DS-FIM-Eval Models fill in missing code between a given prefix and suffix, as in editor plugin completion. | DeepSeek | Function-level problems | University | Dropped | | 1 | Sep 2024 | Sep 2024 | |
| DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap. | Tencent AI Lab and University of Texas at Dallas (Jing et al.) | Data science and SQL | Expert | Dropped | ChatGPT agent 89.9% | 1 | Sep 2024 | Jul 2025 | |
| EgoLife (with memory) Models answer life-assistant questions about a week of egocentric video from six people, such as recalling past events and habits. | EgoLife team (Jingkang Yang, Ziwei Liu et al.) | Long videos | Everyday | In use | | 1 | Mar 2025 | Aug 2026 | |
| Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. | MiniMax | Finance | Expert | Fading | | 1 | Feb 2026 | Feb 2026 | |
| FinSearchComp Global Agents search for financial information about global assets across three task types. The score is the published average accuracy for the Global subset. | ByteDance Seed | Finance | Expert | Dropped | | 1 | Sep 2025 | Oct 2025 | |
| FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. | StepFun | Finance | Expert | In use | | 1 | Sep 2026 | Sep 2026 | |
| Gaokao Math 2026 Models sit the five 2026 Chinese Gaokao mathematics papers (National I, National II, Beijing, Shanghai and Tianjin). Each paper is graded out of 150 points. | iFlytek (SparkLLM) | Competition problems with answers | School | In use | | 1 | Sep 2026 | Oct 2026 | |
| GDPval-Rubrics (MiniMax) MiniMax rubric grading of model deliverables on public GDPval tasks, in an environment matched to GDPval-AA. | MiniMax | Tasks from real jobs | Expert | In use | | 1 | Jun 2026 | Jun 2026 | |
| HaluEval Models judge whether answers, dialogue turns or summaries contain hallucinations. Hallucinated samples were generated by ChatGPT from seed datasets such as HotpotQA and then filtered. | RUCAIBox (Renmin University of China) | Hallucination rates | Everyday | Fading | | 1 | May 2023 | Mar 2026 | |
| Hy-BrowseComp-Pro2 Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published. | Tencent Hy team | Deep research | Expert | In use | | 1 | Aug 2026 | Aug 2026 | |
| Hy-Euler pro Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published. | Tencent Hy team | Research mathematics | Expert | In use | | 1 | Jul 2026 | Jul 2026 | |
| Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. | Tencent Hy team | Finance | Expert | In use | | 1 | Aug 2026 | Aug 2026 | |
| Hy-LifeSearch Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published. | Tencent Hy team | Deep research | Expert | In use | | 1 | Aug 2026 | Aug 2026 | |
| Hy-Math Models solve Tencent in-house math problems without tools. Task and scoring details are not published. | Tencent Hy team | Competition problems with answers | Expert | In use | | 1 | Jul 2026 | Jul 2026 | |
| Hy-SkillsWorld Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published. | Tencent Hy team | General assistants | Expert | In use | | 1 | Jul 2026 | Jul 2026 | |
| Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published. | Tencent Hy team | Building whole projects | Expert | Fading | | 1 | Apr 2026 | Apr 2026 | |
| ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. | Z.ai | Deep research | Expert | Fading | GLM-5V-Turbo 30.7% | 1 | Apr 2026 | Apr 2026 | |
| Infinite Generations (Mistral internal) Models answer challenging long or repetitive prompts from Mistral's internal set. The score is the share of responses that never stop or loop, so lower is better. | Mistral AI | Overall chat quality | Everyday | Dropped | | 1 | Jun 2025 | Jun 2025 | |
| IntPhys 2 Models watch simulated videos and judge whether each event is physically possible, testing object permanence, immutability, continuity and solidity. Scored by accuracy. | Meta FAIR | Physical prediction | Everyday | In use | | 1 | Jun 2025 | Oct 2026 | |
| KAT Claw Bench Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables. | Kuaishou (KwaiKAT team) | General assistants | University | In use | | 1 | Jul 2026 | Jul 2026 | |
| KernelBench Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. | Stanford Scaling Intelligence | GPU kernels | Expert | In use | | 1 | Dec 2024 | Jun 2026 | |
| Longform Writing Models write long-form text without tools, capped at 32k output tokens. Moonshot AI has not published the tasks or the scoring method. | Moonshot AI | Creative writing | University | Dropped | Kimi K2 Thinking 73.8% | 1 | Nov 2025 | Nov 2025 | |
| LSMDC Models fill in a missing word in movie audio descriptions using the matching video clip. Scoring is accuracy. | Tegan Maharaj et al. (LSMDC challenge) | Video understanding | Everyday | Dropped | GPT-4 45.7% | 1 | Nov 2016 | Mar 2023 | |
| LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. | Zhipu AI and Tsinghua University (LVBench), Qwen (memory setup) | Long videos | Everyday | In use | | 1 | Aug 2026 | Aug 2026 | |
| MEWC Models solve 179 Microsoft Excel World Championship problems from 2021 to 2026 with Excel tools. Output cells are compared one by one with answer cells. | MiniMax | Documents and spreadsheets | Competition | Fading | | 1 | Feb 2026 | Feb 2026 | |
| MMSearch Multimodal agents search and answer questions that require visual and textual evidence. | Z.ai | Deep research | Expert | Fading | GLM-5V-Turbo 72.9% | 1 | Apr 2026 | Apr 2026 | |
| Mobile Actions Model converts natural-language phone requests into on-device function calls (calendar, email, maps, flashlight, Wi-Fi). Call accuracy. | Function calling | Everyday | Fading | Reka Edge 88.4% | 1 | Dec 2025 | Mar 2026 | ||
| Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. | Google DeepMind | Embodied reasoning | Everyday | In use | | 1 | Jul 2026 | Jul 2026 | |
| MPDocBench-Parse Models parse complete multipage documents while preserving content and cross-page structure. Evaluation scores text, formulas, tables, figure extraction, reading order, and heading hierarchy. | Bangbang Zhou and collaborators | Documents, charts and text in images | Expert | In use | | 1 | May 2026 | Oct 2026 | |
| MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. | Cohere | Reasoning in many languages | Competition | Fading | | 1 | May 2026 | May 2026 | |
| MultiChallenge Audio (OpenAI) Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy. | OpenAI | Audio understanding and voice | Everyday | Dropped | gpt-realtime 30.5% | 1 | Aug 2025 | Aug 2025 | |
| OpenScore String Quartets Models transcribe images of string quartet sheet music into symbolic notation. Scored as 1 minus normalized edit distance from the reference encoding. | OpenAI, using the OpenScore String Quartets corpus | Documents, charts and text in images | University | In use | GPT-6 Astra 0.84 | 1 | Sep 2026 | Sep 2026 | |
| PAgentBench Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house. | iFlytek (SparkLLM) | General assistants | Everyday | In use | | 1 | Oct 2026 | Oct 2026 | |
| Past IMO (Seed-Prover formal set) Provers write Lean proofs for 155 past IMO problems, mostly adapted from Compfiles and MiniF2F. | ByteDance Seed | Formal proofs | Competition | Dropped | | 1 | Jul 2025 | Jul 2025 | |
| Pile-test Models predict text from the test split of the Pile, an 800GB corpus of 22 sources. Scored by bits per byte. | EleutherAI | Other | Dropped | | 1 | Dec 2020 | Dec 2024 | ||
| ProactiveVideoQA Models answer spoken questions when a streamed video first reveals the answer. | Thinking Machines Lab | Video understanding | Everyday | Fading | | 1 | May 2026 | May 2026 | |
| ProdBench (Tencent) Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3. | Tencent Hy team | General assistants | Expert | In use | | 1 | Jul 2026 | Jul 2026 | |
| QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. | Qwen Team, Alibaba Group | Knowledge in many languages | University | Fading | | 1 | Apr 2026 | Apr 2026 | |
| Readability (Perplexity) Models answer search questions. Graders rate whether answers are concise yet detailed and use markdown formatting well. | Perplexity | Taste and style | Everyday | Dropped | | 1 | Feb 2025 | Feb 2025 | |
| Real-World Audio Recording Speech transcription systems are scored by word error rate on real-world recordings across multiple languages. | OpenAI | Speech recognition and translation | Everyday | In use | GPT-Live-Transcribe 9.60% | 1 | Jul 2026 | Jul 2026 | |
| Real5-OmniDocBench Models parse the same 1,355 document pages under five physical capture conditions. Text edit distance, formula matching, and table similarity determine the overall score. | Baidu PaddlePaddle | Documents, charts and text in images | Expert | In use | | 1 | Mar 2026 | Oct 2026 | |
| ReCoRD Models fill in a masked entity in a query about a news article, using the article as context. Scoring is exact match and token F1. | Johns Hopkins University and Microsoft Research | Reading comprehension | Everyday | Dropped | PaLM 2-L 93.8% | 1 | Oct 2018 | May 2023 | |
| RepCount-A Models count repeated actions in streamed video and are scored by distance from the true count. | Thinking Machines Lab | Video understanding | Everyday | Fading | | 1 | May 2026 | May 2026 | |
| RepoBench Models complete code using context retrieved from other files in the same Python or Java repository. Three tasks cover retrieval, completion and the full pipeline. | UC San Diego (Tianyang Liu, Canwen Xu, Julian McAuley) | Fixing real repositories | Expert | Dropped | | 1 | Jun 2023 | Jan 2025 | |
| SIFo Models follow a chain of dependent instructions, where each step uses the output of the one before. Only the final answer is checked, across 800 samples in four tasks. | University of Amsterdam and University of Groningen (Chen et al.) | Following precise instructions | Everyday | Dropped | | 1 | Jun 2024 | Sep 2025 | |
| Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses. | StepFun | Fixing real repositories | Expert | Fading | | 1 | May 2026 | May 2026 | |
| Story Cloze Winter 2018 Systems select the correct story ending on the Winter 2018 test set. CodaLab reports percentage accuracy under submission identities. | University of Rochester | Commonsense | Everyday | Not in cards | Jan 2018 | ||||
| StoryCloze Spring 2016 Replaced in Jan 2018 | Dropped | PaLM 2-L 87.4% | 1 | Apr 2016 | May 2023 | ||||
| SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. | Ant Group and Peking University | Research science | Expert | In use | | 1 | Oct 2026 | Oct 2026 | |
| SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs. | MiniMax | Web and app front ends | University | In use | | 1 | Jun 2026 | Jun 2026 | |
| SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail. | Shanghai Jiao Tong University | Fixing real repositories | Expert | Dropped | Kimi K2 66.6% | 1 | May 2025 | Sep 2025 | |
| SWebBench Agents browse the web to retrieve information and do research tasks. iFlytek built it in house. | iFlytek (SparkLLM) | Deep research | Expert | In use | | 1 | Oct 2026 | Oct 2026 | |
| TAT-DQA Models answer questions about financial report pages that mix tables and text, often needing arithmetic. Scored by exact match and F1. | NExT++ Research Centre, National University of Singapore | Documents, charts and text in images | University | Dropped | | 1 | Jul 2022 | May 2024 | |
| TheAgentCompany Agents do work tasks in a simulated software company, using a browser, code, and chat with simulated coworkers. Scored by full and partial checkpoint completion. | Carnegie Mellon University et al. | Simulated companies | University | Dropped | 1 | Dec 2024 | Oct 2025 | ||
| TimeSpeak Models initiate speech at specified times while producing the requested content. | Thinking Machines Lab | Speech recognition and translation | Everyday | Fading | | 1 | May 2026 | May 2026 | |
| TVQA Models answer 152,545 multiple-choice questions about clips from 6 TV shows, using video frames and subtitles. Scoring is accuracy. | UNC Chapel Hill | Video understanding | Everyday | Dropped | GPT-4 87.3% | 1 | Sep 2018 | Mar 2023 | |
| UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. | ByteDance Seed | Video games | Everyday | Dropped | | 1 | Sep 2025 | Sep 2025 | |
| VCSUM Models summarize transcripts of 239 real Chinese meetings that total over 230 hours. Scored with ROUGE against human-written summaries. | Han Wu, Mingjie Zhan, Linqi Song, and colleagues | Reasoning over long documents | Everyday | Dropped | | 1 | May 2023 | Sep 2023 | |
| Video-Index Models answer 840 multiple-choice questions about videos, the 210 hardest in each of four groups: perception, time, space and reasoning. Questions that shortcut attacks could answer without the video were left out. | Song et al. (arXiv 2610.00960) | Video understanding | Everyday | In use | | 1 | Oct 2026 | Nov 2026 | |
| VideoHallucer Paired basic and hallucinated questions about videos detect intrinsic and extrinsic hallucinations. Scored by pair accuracy. | BIGAI + Peking University | Video understanding | Everyday | Fading | Reka Edge 59.6% | 1 | Jun 2024 | Mar 2026 | |
| VisualProbe Models search high-resolution images for small targets to answer questions, often over many crop and zoom turns. Answers are checked against a key. | Mini-o3 team (HKU and ByteDance) | Detection and captioning | Everyday | Fading | | 1 | Sep 2025 | May 2026 | |
| WebQuestions Models answer factual questions mined from Google Suggest, with answers taken from the Freebase knowledge base. Scored by exact match. | Stanford University | Trivia and open questions | Everyday | Dropped | PaLM 2-L 28.2% | 1 | Oct 2013 | May 2023 | |
| WiC Models decide whether a target word has the same meaning in two sentence contexts. Scoring is binary accuracy. | Mohammad Taher Pilehvar and Jose Camacho-Collados | Language reasoning | Everyday | Dropped | PaLM 2-L 66.8% | 1 | Aug 2018 | May 2023 | |
| Wild-OmniDocBench Models parse 1,350 photographed document pages with geometric distortion, lighting changes, and other capture artifacts. OmniDocBench metrics score text, formulas, tables, and reading order. | Gengluo Li and collaborators | Documents, charts and text in images | Expert | In use | | 1 | Mar 2026 | Oct 2026 | |
| Winograd Models pick which noun an ambiguous pronoun refers to in Winograd schema sentence pairs that need world knowledge. Scored by accuracy. | Hector Levesque, Ernest Davis, and Leora Morgenstern | Commonsense | Everyday | Dropped | PaLM 2-L 89.5% | 1 | May 2023 | ||
| Work360 Agents complete professional tasks drawn from a map of 26 industries, 182 subfields and 721 specialties. Qwen in-house scoring (method unpublished). | Alibaba Qwen | Tasks from real jobs | Expert | In use | | 1 | Nov 2026 | Nov 2026 | |
| XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. | XBOW | Capture-the-flag challenges | Expert | Fading | Claude Opus 4.7 98.5% | 1 | Aug 2024 | Apr 2026 | |
| XNLI Models classify entailment, contradiction or neutral for sentence pairs translated from MultiNLI into 15 languages. | Facebook AI Research and NYU | Reasoning in many languages | Everyday | Dropped | | 1 | Sep 2018 | Sep 2025 | |
| ZClawBench Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use. | Z.ai (Zhipu AI) | General assistants | Expert | Fading | GLM-5V-Turbo 57.6% | 1 | Mar 2026 | Apr 2026 | |
| BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. | Stanford Vision and Learning Lab | Simulated robots | Everyday | Not in cards | Jul 2026 | ||||
| BEHAVIOR Challenge 2025 Replaced in Jul 2026 | Stanford (BEHAVIOR team) | Fading | 1 | Sep 2025 | Dec 2025 | ||||
| Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy. | Windsurf (Codeium, now part of Cognition) | Fixing real repositories | Expert | Dropped | 1 | May 2025 | May 2025 | ||
| End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score. | Windsurf (Codeium, now part of Cognition) | Fixing real repositories | Expert | Dropped | 1 | May 2025 | May 2025 | ||
| IChO 2025 Models solve the written theoretical problems of the 57th International Chemistry Olympiad, held in Dubai in July 2025. Answers are graded against the official marking scheme. | International Chemistry Olympiad (hosted by the UAE Ministry of Education) | Science olympiads | Competition | Fading | 1 | Jul 2025 | Feb 2026 | ||
| M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. | DAMO Academy, Alibaba Group | Knowledge in many languages | School | Dropped | 1 | Jun 2023 | May 2024 | ||
| Objectron Models predict 3D bounding boxes of objects in short object-centric videos across nine categories. Scored by 3D intersection over union. | Google Research | Detection and captioning | Everyday | Dropped | 1 | Dec 2020 | Sep 2025 | ||
| RGB Models answer English and Chinese news questions from retrieved passages, testing noise robustness, negative rejection, information integration, and counterfactual robustness. Scored by accuracy. | Chinese Academy of Sciences (ISCAS) | Faithful to sources | Everyday | Dropped | 1 | Sep 2023 | Feb 2024 | ||
| BBQ (Bias Benchmark for QA) Ambiguous and disambiguated questions probing social stereotypes across nine attributes. Reports accuracy and bias scores. | NYU MLL | Bias, fairness and politics | System cards only | | 45 | Oct 2021 | Nov 2026 | ||
| Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. | OpenAI with Gryphon Scientific | Lab protocols | Expert | System cards only | GPT-6 Luna 94% | 39 | Aug 2024 | Oct 2026 | |
| Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. | OpenAI with SecureBio | Lab protocols | Expert | System cards only | GPT-Rosalind-6 63.1% | 39 | Dec 2024 | Nov 2026 | |
| OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. | OpenAI | Refusing harmful requests | System cards only | 35 | Nov 2026 | ||||
| Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. | OpenAI | Refusing harmful requests | Everyday | System cards only | GPT-4.5 99% | 33 | Dec 2024 | Nov 2026 | |
| TroubleshootingBench Models find and correct realistic experimental errors in expert-written wet lab protocols. Scored as pass@1. | OpenAI | Lab protocols | Expert | System cards only | GPT-Rosalind-6 59% | 29 | Aug 2025 | Oct 2026 | |
| OpenAI cyber safety evaluations (incl. Daybreak Blue) Refusal and helpfulness evaluations for cyber requests, including the Daybreak Blue set for enabling defensive security work. | OpenAI | Refusing harmful requests | System cards only | 24 | Nov 2026 | ||||
| OpenAI Dynamic Mental Health Benchmarks Multi-turn conversations with adversarial simulated users on mental-health risks. Scored on safe and supportive responses. | OpenAI | Wellbeing and sycophancy | System cards only | 22 | Nov 2026 | ||||
| Multiturn Jailbreak Evaluations Models face adaptive multi-turn jailbreak attackers with growing attack budgets. Scored as worst-case defender success rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-6.2 Astra 80.9% | 21 | Mar 2026 | Nov 2026 | |
| CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. | UIUC (Kang Lab) | Finding and exploiting real bugs | Expert | Not in cards | Oct 2025 | ||||
| CVE-Bench v1 Replaced in Oct 2025 | System cards only | GPT-5.6 Sol 94.1% | 20 | Mar 2025 | Sep 2026 | ||||
| CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. | OpenAI + NYU (Chen Yueh-Han et al.) | Deception and alignment | System cards only | | 20 | Mar 2026 | Nov 2026 | ||
| Single-turn benign request evaluations (Anthropic) Models answer benign requests on sensitive topics. Score is the refusal rate, where lower is better. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Fable 5.1 0% | 19 | May 2025 | Nov 2026 | |
| Single-turn harmful request evaluations (Anthropic) Models answer single-turn requests that violate the usage policy. Score is the harmless response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Opus 4.5 99.8% | 19 | May 2025 | Nov 2026 | |
| XSTest Models answer 250 safe prompts that resemble unsafe ones and 200 unsafe contrast prompts. Labs usually report the incorrect refusal rate on safe prompts. | Röttger et al. (Bocconi University and others) | Refusing harmful requests | Everyday | System cards only | | 19 | Aug 2023 | Mar 2026 | |
| Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. | Anthropic | Deception and alignment | System cards only | Claude Fable 5 2.06 | 19 | Nov 2026 | |||
| OpenAI user-flagged hallucination eval Production conversations users flagged for factual errors. Measures the rate of incorrect claims. | OpenAI | Hallucination rates | System cards only | GPT-5.5 9.20% | 19 | Nov 2026 | |||
| Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-6 Cyber 99.9% | 19 | Jun 2026 | Nov 2026 | |
| Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Sonnet 4.6 99.4% | 18 | Feb 2025 | Nov 2026 | |
| Multi-turn testing (Anthropic) Models face multi-turn conversations that escalate toward harmful requests across risk areas. Score is the appropriate response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | 18 | May 2025 | Nov 2026 | ||
| SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. | Anthropic + Scale AI + Redwood (Kutasov et al.) | Deception and alignment | System cards only | Claude Opus 5.5 26.2% | 17 | Jun 2025 | Oct 2026 | ||
| Standard Refusal Evaluation Models answer disallowed and benign safety-related prompts across policy categories. An autograder scores not_unsafe and not_overrefuse. | OpenAI | Refusing harmful requests | Everyday | System cards only | o1 100% | 17 | Aug 2024 | Aug 2025 | |
| Child safety evaluations (Anthropic) Models answer single-turn and multi-turn prompts about child sexualization and grooming. Score is the appropriate response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | 17 | Feb 2025 | Nov 2026 | ||
| Malicious use of Claude Code (Anthropic) Agents in Claude Code receive malicious and dual-use coding requests. Score is the refusal rate on malicious requests. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Mythos Preview 96.7% | 16 | Sep 2025 | Nov 2026 | |
| Tone An automated evaluation measures whether model refusals are objective, neutral and not preachy. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 16 | Apr 2025 | Oct 2026 | ||
| LABBench2 SeqQA2 Models reason about DNA and protein sequences and answer in open response. The SeqQA2 subtask of LABBench2. | Edison Scientific (formerly FutureHouse) | Genomics and life sciences | Expert | Not in cards | | Feb 2026 | Feb 2026 | ||
| LAB-Bench SeqQA v1 Replaced in Feb 2026 | FutureHouse | System cards only | | 15 | Jul 2024 | Oct 2026 | |||
| Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 15 | Apr 2025 | Oct 2026 | ||
| Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 15 | Apr 2025 | Oct 2026 | ||
| Text to Text Safety An automated content safety evaluation measures policy violations in model answers to text prompts. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 15 | Apr 2025 | Oct 2026 | ||
| Gray Swan Agent Red Teaming (ART) benchmark Agents face curated prompt injections from a large public red-teaming competition across 44 deployment scenarios. Score is attack success rate. | Gray Swan + UK AI Security Institute | Jailbreaks and prompt injection | Expert | System cards only | Claude Mythos Preview 0.10% | 14 | Jul 2025 | Oct 2026 | |
| Anthropic political even-handedness eval Models answer paired prompts that take opposing sides on 150 US political topics. A grader scores even-handedness, refusals and acknowledgment of opposing views. | Anthropic | Bias, fairness and politics | Everyday | System cards only | Claude Opus 5.5 99.4% | 14 | Nov 2025 | Nov 2026 | |
| Prompt injection robustness against adaptive attackers, coding (Anthropic) Coding agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 4.6 0% | 14 | Nov 2025 | Nov 2026 | |
| Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 4.5 0% | 14 | Nov 2025 | Nov 2026 | |
| Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. | OpenAI | Capture-the-flag challenges | University | System cards only | o3 68% | 14 | Aug 2024 | Aug 2025 | |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | OpenAI | Finding and exploiting real bugs | Expert | System cards only | GPT-5.5 93.3% | 14 | Apr 2025 | Aug 2026 | |
| Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-6.2 Astra 81.8% | 14 | Apr 2026 | Nov 2026 | |
| ToxiGen Models continue ToxiGen prompts about 13 minority groups. Scored by the average toxicity of the generations. | Microsoft Research + MIT (Hartvigsen et al.) | Bias, fairness and politics | Everyday | System cards only | | 14 | Mar 2022 | Dec 2025 | |
| Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. | OpenAI with Gryphon Scientific | Hazardous knowledge | Expert | System cards only | OpenAI deep research 0% | 14 | Sep 2024 | Nov 2025 | |
| Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude 3.7 Sonnet 69.7% | 14 | Feb 2025 | Jul 2026 | |
| Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Sonnet 5.5 0% | 13 | Nov 2025 | Oct 2026 | |
| OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-5.2 55% | 13 | Jan 2025 | Dec 2025 | |
| First-Person Fairness evaluation Tests whether a user's name (a demographic proxy) changes responses, using an LLM research assistant to detect harmful stereotypes. | OpenAI | Bias, fairness and politics | System cards only | ChatGPT agent 0.40% | 13 | Oct 2024 | Jul 2026 | ||
| KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. | OpenAI | GPU kernels | Research frontier | System cards only | GPT-6.2 Astra 76.9% | 13 | Jun 2026 | Nov 2026 | |
| DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4.7 80% | 13 | May 2025 | Jul 2026 | |
| Suicide and self-harm evaluations (Anthropic) Models handle single-turn and multi-turn conversations about suicide and self-harm. Score is the appropriate response rate. | Anthropic | Wellbeing and sycophancy | Everyday | System cards only | 13 | Sep 2025 | Nov 2026 | ||
| Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Opus 5 31.3 | 12 | Feb 2025 | Jul 2026 | |
| WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. | CAIS + Scale AI et al. | Hazardous knowledge | Expert | System cards only | GPT-6 Astra 84.8% | 12 | Mar 2024 | Sep 2026 | |
| LLM training task (Anthropic) Agents optimize a CPU-only small language model training implementation. Scored by speedup. | Anthropic | Machine learning engineering | Expert | System cards only | 12 | Feb 2025 | Jul 2026 | ||
| U18 safety evaluations Models answer challenging prompts in teen safety categories such as age-restricted goods, eating disorders and emotional reliance. Scored as safe completion rate per category. | OpenAI | Refusing harmful requests | Everyday | System cards only | 12 | Aug 2026 | Nov 2026 | ||
| Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. | Anthropic | Deception and alignment | Expert | System cards only | Claude Haiku 4.5 30% | 11 | May 2025 | Sep 2026 | |
| Time series forecasting task (Anthropic) Agents solve a regression and time-series forecasting problem with known state-of-the-art results, in six variants. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Mythos Preview 4.55 | 11 | May 2025 | Jul 2026 | |
| Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. | OpenAI | Deception and alignment | Expert | System cards only | GPT-5.5 90% | 11 | Nov 2025 | Jul 2026 | |
| Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5.6 Sol 100% | 11 | Dec 2025 | Aug 2026 | |
| Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-6 Astra 99.8% | 11 | Jul 2026 | Oct 2026 | |
| Novel compiler task (Anthropic) Agents build a compiler for an unusual programming language from a specification and test cases. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Opus 5 80.9% | 11 | May 2025 | Jul 2026 | |
| Disordered eating evaluations (Anthropic) Models handle conversations about eating disorders and disordered eating. Score is the appropriate response rate. | Anthropic | Wellbeing and sycophancy | Everyday | System cards only | 11 | Feb 2026 | Nov 2026 | ||
| Kernels task (Anthropic) Agents optimize compute kernels in a performance engineering challenge. Scored by speedup. | Anthropic | GPU kernels | Expert | System cards only | 11 | May 2025 | Jul 2026 | ||
| Human Pathogen Capabilities Test v2 (HPCT v2) Models answer multiple-response questions on a small set of weaponizable pathogens that biosecurity experts rank highest priority. A revised version of HPCT. | SecureBio | Hazardous knowledge | Research frontier | Not in cards | Claude Opus 5 80% | 2026 | Oct 2026 | ||
| Human Pathogen Capabilities Test v1 Replaced in Jan 2026 | Expert | System cards only | Claude Opus 4.6 80.7% | 10 | Aug 2025 | Oct 2026 | |||
| AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6.2 Astra 0.59 | 10 | Jun 2026 | Nov 2026 | |
| Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. | OpenAI | Capture-the-flag challenges | School | System cards only | OpenAI deep research 82% | 10 | Aug 2024 | Jul 2025 | |
| OpenAI Research Engineer Interview: Multiple Choice Models answer 97 multiple-choice questions drawn from OpenAI machine learning interview topics. Scored as cons@32. | OpenAI | Labs' internal R&D tests | Expert | System cards only | o4-mini 83% | 10 | Aug 2024 | Jul 2025 | |
| Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6.1 Terra 0.47 | 10 | Sep 2026 | Nov 2026 | |
| GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. | Google DeepMind | Deception and alignment | Expert | System cards only | | 10 | May 2025 | Oct 2026 | |
| Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6 Luna 10.7 | 10 | Sep 2026 | Nov 2026 | |
| SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6.2 Astra 0.46 | 10 | Sep 2026 | Nov 2026 | |
| Instruction hierarchy attacks Models face attacks that try to override higher-priority instructions. Scored as defender success rate averaged per defender query. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-6 Astra 99.99% | 10 | Jul 2026 | Oct 2026 | |
| GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. | Google DeepMind | Deception and alignment | Expert | System cards only | | 10 | May 2025 | Oct 2026 | |
| Self-preference bias (Anthropic) Models grade transcripts attributed to themselves or to other models. Measures whether scores favor the model's own outputs. | Anthropic | Bias, fairness and politics | Everyday | System cards only | Claude Opus 5.5 0.07 | 10 | Sep 2025 | Oct 2026 | |
| User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. | OpenAI | Refusing harmful requests | Everyday | System cards only | OpenAI Computer-Using Agent (CUA) 92% | 10 | Jan 2025 | Jul 2026 | |
| Election integrity evaluations (Anthropic) Models answer election-related prompts, including requests for misinformation or influence operations. Score is the appropriate response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | 10 | Oct 2024 | Nov 2026 | ||
| Molecular Biology Capabilities Test Models answer expert-level molecular biology questions from SecureBio. Scored as accuracy. | SecureBio | Genomics and life sciences | Expert | System cards only | Claude Opus 5.5 68.7% | 9 | Oct 2026 | ||
| BioLP-bench Find the introduced error in a biological lab protocol. Tests protocol understanding. | Igor Ivanov (Oxford Biosecurity Group) | Lab protocols | Expert | System cards only | OpenAI deep research 49% | 9 | Aug 2024 | Nov 2025 | |
| Challenging Refusal Evaluation Models answer a harder set of disallowed content prompts across policy categories. An autograder scores the not_unsafe rate. | OpenAI | Refusing harmful requests | Expert | System cards only | o1-preview 93.4% | 9 | Sep 2024 | Apr 2025 | |
| OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-5.1-Codex-Max 8.00% | 9 | Aug 2025 | Apr 2026 | |
| PersonQA Models answer questions about publicly available facts on people. Scored on accuracy, with hallucination rate reported alongside. | OpenAI | Hallucination rates | Everyday | System cards only | ChatGPT agent 94.3% | 9 | Dec 2024 | Aug 2025 | |
| SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Sonnet 4.6 48.2% | 9 | Feb 2025 | Feb 2026 | |
| 100Q-Hard (Anthropic) Models answer hard, obscure factual questions. Scored on correct, incorrect and declined answers. | Anthropic | Short factual questions | Expert | System cards only | Claude Mythos Preview 60.1% | 9 | Mar 2024 | May 2026 | |
| OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. | OpenAI | Deception and alignment | System cards only | GPT-6.2 Sol 0.47% | 9 | Nov 2026 | |||
| OpenAI Research Engineer Interview: Coding Models solve 18 self-contained coding problems like those in OpenAI interviews. Scored as pass@1. | OpenAI | Labs' internal R&D tests | Expert | System cards only | o1 100% | 9 | Aug 2024 | Apr 2025 | |
| NanoGPT Speedrun Improve a small GPT training script to reach a target validation loss as fast as possible on fixed hardware. | Keller Jordan (community), OpenAI and METR (variants) | Machine learning engineering | Research frontier | System cards only | GPT-6.2 Astra 42.9% | 9 | Jun 2024 | Nov 2026 | |
| Instruction Hierarchy Evaluation Models resolve conflicts between system, developer and user messages, resist tutor jailbreaks, and protect secret phrases and passwords. Scored per sub-evaluation. | OpenAI | Jailbreaks and prompt injection | Everyday | System cards only | 9 | Dec 2024 | Aug 2025 | ||
| Malicious agentic influence campaigns, 34-point rubric (Anthropic) Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale. | Anthropic | Dangerous capabilities | Expert | System cards only | 1 | Nov 2026 | Nov 2026 | ||
| Malicious agentic influence campaigns (Anthropic) Original Replaced in Nov 2026 | Refusing harmful requests | Everyday | System cards only | 8 | Apr 2026 | Sep 2026 | |||
| Unjustified refusals An automated evaluation measures over-refusals on benign or borderline prompts that can be safely fulfilled. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 9 | Oct 2026 | |||
| Human Sourced Jailbreaks Models face jailbreak prompts collected from human red teaming. Scored as the not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | OpenAI Computer-Using Agent (CUA) 100% | 8 | Sep 2024 | Apr 2025 | |
| MLE-bench 30 Agents train models for 30 of MLE-bench's Kaggle-style competitions. The score is the share of runs that earn a bronze medal or better. | OpenAI | Machine learning engineering | Expert | System cards only | GPT-5.5 36.7% | 8 | Aug 2025 | Jun 2026 | |
| RACE Models answer reading comprehension questions from Chinese middle and high school English exams. Reported here without a subset split. | Carnegie Mellon University | Reading comprehension | School | System cards only | | 8 | Apr 2017 | Oct 2026 | |
| Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.5 0% | 8 | May 2025 | Feb 2026 | |
| Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Luna 0% | 8 | Sep 2026 | Nov 2026 | |
| Agentic coding malicious requests (Anthropic) Coding agents with standard coding tools receive 150 malicious coding requests prohibited by policy. Score is the refusal rate. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Haiku 4.5 100% | 8 | May 2025 | Feb 2026 | |
| Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. | OpenAI | Labs' internal R&D tests | Expert | System cards only | OpenAI deep research 78% | 8 | Aug 2024 | Feb 2025 | |
| Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. | Anthropic (with Benchling) | Lab protocols | Expert | System cards only | Claude Sonnet 5 62% | 8 | Apr 2026 | Oct 2026 | |
| Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. | Anthropic (with Dyno Therapeutics) | Hazardous knowledge | Expert | System cards only | 8 | May 2026 | Oct 2026 | ||
| Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. | Anthropic | Deception and alignment | Expert | Not in cards | Jan 2026 | ||||
| Petri 1.0 Replaced in Jan 2026 | System cards only | 8 | Oct 2025 | Jul 2026 | |||||
| METR Time Horizon 2 Agents complete 36 tasks across 12 families. METR estimates how long a skilled human would need for tasks that agents complete with 50 percent reliability. | METR | Task length | Research frontier | Not in cards | GPT-6 Astra 27.1 h | Nov 2026 | Sep 2026 | ||
| METR Time Horizon 1.1 Replaced in Nov 2026 | Not in cards | Claude Mythos Preview 17.4 h | Jan 2026 | Apr 2026 | |||||
| METR Time Horizon 1.0 Replaced in Jan 2026 | System cards only | Claude Opus 4.5 4.1 h | 7 | Mar 2025 | Jul 2026 | ||||
| CharXiv (DQ) Models answer descriptive questions about charts from arXiv papers, such as titles, axis labels and legend entries. It is the descriptive half of CharXiv. | Princeton University (Zirui Wang et al.) | Documents, charts and text in images | University | System cards only | o3 95% | 7 | Jun 2024 | Jun 2026 | |
| scBench (LatchBio SingleCellBench) 195 verifiable single-cell RNA-seq analysis problems. Each pairs a data snapshot with a task and a deterministic grader. | LatchBio | Lab and data work | Expert | System cards only | GPT-6 Astra 63.9% | 7 | Feb 2026 | Oct 2026 | |
| SpatialBench (LatchBio) Agentic spatial-transcriptomics data analysis problems with deterministic graders. Anthropic reports a 115-problem Verified variant. | LatchBio | Lab and data work | Expert | System cards only | GPT-5.5 57.6% | 7 | Dec 2025 | Oct 2026 | |
| MentalHealthBench 1,215 synthetic mental-health conversations with 5,262 clinician-authored rubric criteria. An LLM judge scores safety and guidance quality. | OpenAI | Wellbeing and sycophancy | Expert | System cards only | GPT-6 Astra 57.3% | 7 | Sep 2026 | Nov 2026 | |
| Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4.6 60.3% | 7 | May 2025 | Feb 2026 | |
| Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Opus 4.5 0.97 | 7 | May 2025 | Feb 2026 | |
| Anthropic ECI (AECI) Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. | Anthropic | Composite indices | Research frontier | System cards only | Claude Opus 5.5 169.4 | 7 | Apr 2026 | Oct 2026 | |
| Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 94.4% | 7 | Feb 2025 | Feb 2026 | |
| Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 100% | 7 | Feb 2025 | Feb 2026 | |
| Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.6 71.4% | 7 | Feb 2025 | Feb 2026 | |
| Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 100% | 7 | Feb 2025 | Feb 2026 | |
| Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.6 100% | 7 | Feb 2025 | Feb 2026 | |
| LSAT Models take a simulated Law School Admission Test. Scored on the 120 to 180 scale. | LSAC | Language reasoning | University | System cards only | GPT-4 163.0 | 7 | Nov 2024 | ||
| Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 0% | 7 | Sep 2026 | Nov 2026 | |
| BOLD Models continue 23,679 English Wikipedia prompts about professions, gender, race, religion and politics. Classifiers score sentiment, toxicity and polarity per group. | Amazon | Bias, fairness and politics | Everyday | System cards only | | 7 | Jan 2021 | Mar 2024 | |
| Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. | OpenAI | Refusing harmful requests | Everyday | System cards only | 7 | Sep 2026 | Nov 2026 | ||
| CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. | OpenAI | Deception and alignment | Expert | System cards only | 7 | Mar 2026 | Nov 2026 | ||
| Discrim-Eval Models make yes or no decisions in templated scenarios that vary age, gender and race. Scored by mixed effects model coefficients. | Anthropic (Tamkin et al.) | Bias, fairness and politics | Everyday | System cards only | 7 | Dec 2023 | Jan 2025 | ||
| Instruction Following (safety evaluation) An automated evaluation measures whether the model follows instructions on sensitive prompts while staying within safety policies. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 7 | Sep 2025 | |||
| Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. | Anthropic | Hazardous knowledge | Expert | System cards only | 7 | May 2025 | Feb 2026 | ||
| LinuxArena AI-control setting in live Linux production-like environments. Measures stealthy side-task completion against action and CoT monitors. | Redwood Research + EquiStamp | Deception and alignment | Expert | System cards only | Claude Sonnet 5 0% | 6 | Apr 2026 | Oct 2026 | |
| Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. | Anthropic | Refusing harmful requests | System cards only | Claude Opus 5 1.13 | 6 | May 2026 | Oct 2026 | ||
| Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. | Anthropic | Finding and exploiting real bugs | Expert | System cards only | Claude Opus 5.5 91.1% | 6 | Oct 2026 | ||
| Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Sol 1.30% | 6 | Aug 2025 | Sep 2026 | |
| False-premise questions (Anthropic) Models answer questions built on false premises. Measures how often they accept the premise instead of correcting it. | Anthropic | Hallucination rates | Everyday | System cards only | Claude Opus 4.5 1.20% | 6 | Sep 2025 | May 2026 | |
| GRE Quantitative Models take the GRE quantitative reasoning section. Scored on the 130 to 170 scale. | ETS | Word problems | University | System cards only | GPT-4 163.0 | 6 | Nov 2024 | ||
| GRE Verbal Models take the GRE verbal reasoning section. Scored on the 130 to 170 scale. | ETS | Reading comprehension | University | System cards only | GPT-4 169.0 | 6 | Nov 2024 | ||
| Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. | Anthropic | AI research tasks | Expert | System cards only | Claude Mythos Preview 0.65 | 6 | May 2025 | Apr 2026 | |
| MakeMePay A con artist model tries to talk a GPT-4o mark into paying money in a role-play. Scored as the share of conversations with a payment. | OpenAI | Dangerous capabilities | Everyday | System cards only | GPT-4.5 57% | 6 | Sep 2023 | Feb 2025 | |
| Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Fable 5.2 3.90% | 6 | Jun 2026 | Nov 2026 | |
| Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. | OpenAI | Finding and exploiting real bugs | Research frontier | System cards only | GPT-6 Astra 45.5% | 6 | Sep 2026 | Nov 2026 | |
| Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Sonnet 4.5 78% | 6 | Oct 2025 | ||
| Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. | Anthropic | Deception and alignment | Expert | System cards only | Claude Fable 5 3.10% | 6 | Feb 2026 | Jun 2026 | |
| DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-5.5 Pro 16.5% | 6 | Apr 2026 | Sep 2026 | |
| Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-5.6 Sol 7.60% | 6 | Apr 2026 | Sep 2026 | |
| Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. | OpenAI | Deception and alignment | Expert | System cards only | GPT-Rosalind-6 1.40% | 6 | Sep 2026 | Nov 2026 | |
| API-Bank Models make API calls in dialogues that need tools such as calendars and search. Scored on call accuracy. | Alibaba DAMO (Li et al.) | Function calling | Everyday | System cards only | | 6 | Apr 2023 | Nov 2024 | |
| Factuality Heavy prompts Models answer prompts representative of factuality-heavy ChatGPT conversations. A grader with web access counts responses and claims with factual errors. | OpenAI | Hallucination rates | Everyday | System cards only | GPT-6 Sol 5.40% | 6 | May 2026 | Oct 2026 | |
| Gorilla APIBench Models write calls to machine learning APIs from Hugging Face, TorchHub and TensorHub. Scored by AST match. | UC Berkeley (Gorilla) | Function calling | Everyday | System cards only | | 6 | May 2023 | Nov 2024 | |
| High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. | OpenAI | Hallucination rates | Expert | System cards only | GPT-6 Sol 4.10% | 6 | May 2026 | Oct 2026 | |
| Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. | Meta | Knowledge in many languages | University | System cards only | | 6 | Jul 2024 | Nov 2024 | |
| Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5.6 Terra 94.6% | 6 | Jun 2026 | Jul 2026 | |
| Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 0.63% | 6 | May 2025 | Sep 2026 | |
| WildChat non-toxic incorrect refusals (Anthropic) Models answer non-toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the incorrect refusal rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude 3.5 Haiku 4.80% | 6 | Mar 2024 | Oct 2024 | |
| WildChat toxic refusals (Anthropic) Models answer toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the correct refusal rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude 3.5 Sonnet 96.4% | 6 | Mar 2024 | Oct 2024 | |
| RealToxicityPrompts Models continue 100,000 web text prompts. Scored on the rate of toxic generations judged by a toxicity classifier. | Allen Institute for AI and University of Washington | Refusing harmful requests | Everyday | System cards only | | 6 | Sep 2020 | Sep 2025 | |
| AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. | Anthropic (with Dyno Therapeutics) | Hazardous knowledge | Expert | System cards only | Claude Opus 4.8 0.80 | 6 | May 2026 | Oct 2026 | |
| Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | | 6 | Nov 2025 | Oct 2026 | |
| OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. | OpenAI | Deception and alignment | Expert | System cards only | GPT-Rosalind-6 0% | 6 | Sep 2026 | Nov 2026 | |
| Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Qwen Team, Alibaba Group | Chat and instructions in many languages | Everyday | System cards only | | 6 | Jul 2024 | Dec 2024 | |
| Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Qwen Team, Alibaba Group | Other | Everyday | System cards only | | 6 | Jul 2024 | Dec 2024 | |
| RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. | METR | AI research tasks | Research frontier | System cards only | | 6 | Sep 2024 | Feb 2026 | |
| CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. | Google DeepMind | Hazardous knowledge | Expert | System cards only | 6 | Nov 2025 | Oct 2026 | ||
| Static Jailbreak Evaluations Models face a fixed set of jailbreak attacks on bio, violence and cyber requests. Scored as defender success rate per category. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | 6 | Sep 2026 | Nov 2026 | ||
| World-Class Bio Models answer expert-level biology questions in a SecureBio benchmark. Scored as accuracy. | SecureBio | Genomics and life sciences | Research frontier | System cards only | Claude Opus 5 68.8% | 5 | Oct 2026 | ||
| ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. | SecureBio | Hazardous knowledge | Expert | System cards only | | 5 | Jun 2026 | Oct 2026 | |
| Belebele Models answer multiple-choice reading comprehension questions on FLORES passages in 122 language variants. Scored by accuracy. | Meta AI (Bandarkar et al.) | Reading in many languages | Everyday | System cards only | GPT-3.5 Turbo 51.1% | 5 | Aug 2023 | Oct 2026 | |
| AMC 12 2024 Models answer the 25 multiple-choice problems of the 2024 AMC 12 contests. Scored on the official 150-point scale. | Mathematical Association of America | Competition problems with answers | Competition | System cards only | | 1 | Nov 2024 | Sep 2025 | |
| AMC 12 2023 Replaced in Nov 2024 | System cards only | Claude 3 Opus 63.0 | 3 | Nov 2023 | Mar 2024 | ||||
| AMC 12 2022 Replaced in Nov 2023 | System cards only | GPT-4 60.0 | 1 | Nov 2022 | Mar 2023 | ||||
| Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. | Anthropic | Deception and alignment | Expert | System cards only | Claude Sonnet 4 14.3% | 5 | Oct 2025 | ||
| Cyber harm coverage (Anthropic) Cyber safeguard classifiers screen examples from each cyber harm category. Score is overall recall, the share of harmful examples flagged. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Fable 5.1 100% | 5 | Jul 2026 | Nov 2026 | |
| Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. | Anthropic | Finding and exploiting real bugs | Research frontier | System cards only | Claude Mythos Preview 72.4% | 5 | Jul 2026 | ||
| Internal model use survey (Anthropic) Anthropic technical staff estimate productivity uplift from the model and judge whether it could automate a junior ML researcher. | Anthropic | Labs' internal R&D tests | Expert | System cards only | Claude Opus 4.5 50% | 5 | May 2025 | Apr 2026 | |
| Jailbreak Augmented Examples Models face publicly known jailbreaks applied to prompts from the standard disallowed content set. Scored as not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Everyday | System cards only | o1-mini 100% | 5 | Sep 2024 | Jan 2025 | |
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) Models continue conversations about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. | Microsoft | Refusing harmful requests | Everyday | System cards only | | 5 | Apr 2024 | Dec 2024 | |
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) Models summarize content about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. | Microsoft | Refusing harmful requests | Everyday | System cards only | | 5 | Apr 2024 | Dec 2024 | |
| Microsoft RAI benchmark: Jailbreak (DR-1) Models face simulated jailbreak conversations on topics such as child grooming, illegal advice and leaking guidelines. Scored as the defect rate at severity 1. | Microsoft | Jailbreaks and prompt injection | Everyday | System cards only | | 5 | Apr 2024 | Dec 2024 | |
| Microsoft RAI benchmark: Third Party Harm (DR-1) Models face simulated conversations that try to extract third-party content such as books, news, recipes and songs. Scored as the defect rate at severity 1. | Microsoft | Refusing harmful requests | Everyday | System cards only | | 5 | Apr 2024 | Dec 2024 | |
| Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-6 Astra 76% | 5 | Feb 2026 | Sep 2026 | |
| Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. | Anthropic | Research science | Expert | System cards only | Claude Opus 4.8 86.2% | 5 | Feb 2026 | Jun 2026 | |
| Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.8 9.40% | 5 | Feb 2026 | May 2026 | |
| Production Jailbreaks Models face jailbreaks found in production ChatGPT data. Scored as not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Everyday | System cards only | OpenAI Computer-Using Agent (CUA) 100% | 5 | Sep 2024 | Jan 2025 | |
| UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. | UK AI Security Institute | Deception and alignment | Expert | System cards only | Claude Opus 4.7 0% | 5 | Apr 2026 | Sep 2026 | |
| WildGuardTest Models classify whether user prompts are harmful, whether responses are harmful, and whether responses refuse, across 13 risk categories. Scored by F1. | Allen Institute for AI | Refusing harmful requests | Everyday | System cards only | | 5 | Jun 2024 | Dec 2025 | |
| Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. | Anthropic | Research science | Expert | System cards only | Claude Opus 4.8 79% | 5 | Feb 2026 | Jun 2026 | |
| Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. | Anthropic | Research science | Expert | System cards only | Claude Opus 5 42.5% | 5 | Jun 2026 | Oct 2026 | |
| Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. | OpenAI with Ranger | Hazardous knowledge | Expert | System cards only | 5 | Sep 2024 | Jan 2025 | ||
| Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. | OpenAI | Deception and alignment | Expert | System cards only | 5 | Jun 2026 | Nov 2026 | ||
| BioTIER Refuse Models receive 398 dangerous biology prompts. The score measures the percentage correctly refused. | SecureBio | Refusing harmful requests | Expert | System cards only | Claude Opus 5.5 99.1% | 4 | Mar 2026 | Nov 2026 | |
| ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. | SecureBio | Hazardous knowledge | Expert | System cards only | | 4 | Jun 2026 | Oct 2026 | |
| SWE-Lancer Diamond (SWE Manager) Models review competing implementation proposals for real freelance tasks and pick the best one. Scored against the original choice as pass@1. | OpenAI | Fixing real repositories | Expert | System cards only | Claude 3.5 Sonnet 44.9% | 4 | Feb 2025 | Apr 2025 | |
| LongFact-Objects Models write detailed answers to LongFact prompts about specific objects such as people or places. A fact checker scores the claim-level error rate. | Google DeepMind | Long-form factuality | Everyday | System cards only | GPT-5 0.80% | 4 | Mar 2024 | Feb 2026 | |
| In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. | Google DeepMind | Capture-the-flag challenges | Expert | System cards only | | 4 | Mar 2024 | Jul 2025 | |
| InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. | Princeton (Yang et al.) | Capture-the-flag challenges | School | System cards only | | 4 | Jun 2023 | Nov 2025 | |
| Where2Place 100 cluttered real photos with free-space placement instructions. The model points, scored correct if inside the ground-truth mask. | University of Washington + NVIDIA (RoboPoint) | Embodied reasoning | Everyday | System cards only | | 4 | Jun 2024 | May 2026 | |
| WildChat (toxic conversations) Models answer toxic user turns from WildChat, the 200 highest scoring conversations per moderation category. Scored as not_unsafe rate. | Allen Institute for AI (dataset), OpenAI (toxic subset) | Refusing harmful requests | Everyday | System cards only | GPT-4.5 98% | 4 | May 2024 | Mar 2025 | |
| AMC 10 2023 Models take the 2023 AMC 10 A and B exams. Scored out of 150 points. | Mathematical Association of America | Competition problems with answers | Competition | System cards only | Claude 3 Opus 72.0 | 3 | Nov 2023 | Mar 2024 | |
| AMC 10 2022 Replaced in Nov 2023 | System cards only | GPT-4 30.0 | 1 | Nov 2022 | Mar 2023 | ||||
| Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5.1-Codex-Max 100% | 4 | May 2025 | Nov 2025 | |
| Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. | OpenAI | Dangerous capabilities | Expert | System cards only | OpenAI deep research 77% | 4 | Dec 2024 | Feb 2025 | |
| CrowS-Pairs Models choose between paired sentences that differ only in a stereotyped social group. Measures preference for the stereotyped sentence. | NYU (Nangia et al.) | Bias, fairness and politics | Everyday | System cards only | | 4 | Sep 2020 | Jul 2024 | |
| Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. | Google DeepMind with Irregular | Capture-the-flag challenges | Expert | System cards only | | 4 | Mar 2025 | Nov 2025 | |
| Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. | Google DeepMind | Capture-the-flag challenges | Expert | System cards only | | 4 | Feb 2024 | Jul 2025 | |
| Higher-difficulty benign request evaluations (Anthropic) Models answer harder benign requests that resemble harmful ones. Score is the refusal rate, where lower is better. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Opus 4.7 0.01% | 4 | Feb 2026 | Apr 2026 | |
| Higher-difficulty harmful request evaluations (Anthropic) Models answer harder, more obfuscated policy-violating requests. Score is the harmless response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Sonnet 4.6 99.4% | 4 | Feb 2026 | Apr 2026 | |
| LongFact-Concepts Models write detailed answers to LongFact prompts about broad concepts. A fact checker scores the claim-level error rate. | Google DeepMind | Long-form factuality | Everyday | System cards only | GPT-5 mini 0.90% | 4 | Mar 2024 | Feb 2026 | |
| MBE (Multistate Bar Examination) Models answer the 200 multiple choice questions of an official Multistate Bar Examination practice exam. Scored by accuracy. | NCBE | Law | Expert | System cards only | Claude 3 Opus 85% | 4 | Mar 2024 | ||
| Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. | Meta + Faculty AI | Genomics and life sciences | Expert | System cards only | | 4 | Oct 2026 | ||
| Meta chemical-agent refusal evaluation Models receive high-risk chemical prompts and the benchmark measures how often they refuse. Covers development, procurement and production stages. | Meta | Hazardous knowledge | Expert | System cards only | | 4 | Oct 2026 | ||
| Microsoft RAI benchmark: Ungroundedness Models answer in GPT-4 simulated multi-turn conversations. A judge rates how ungrounded each answer is in the given prompt, from 0 to 4. | Microsoft | Faithful to sources | Everyday | System cards only | | 4 | Apr 2024 | Aug 2024 | |
| Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. | OpenAI | Dangerous capabilities | Expert | System cards only | OpenAI deep research 74% | 4 | Dec 2024 | Feb 2025 | |
| SecureBio VMQA, single choice Models answer 350 virology troubleshooting questions in single-choice form. VMQA is an earlier version of the Virology Capabilities Test. | SecureBio | Hazardous knowledge | Expert | System cards only | | 4 | Nov 2025 | ||
| UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. | UK AI Security Institute | Finding and exploiting real bugs | Research frontier | System cards only | Claude Opus 5 80% | 4 | Apr 2026 | Jul 2026 | |
| UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. | UK AI Security Institute | Deception and alignment | Expert | System cards only | Claude Opus 4.7 0% | 4 | Apr 2026 | Jul 2026 | |
| Winogender Models resolve pronouns in sentences about occupations. Accuracy shows whether gender stereotypes sway coreference. | Johns Hopkins University (Rudinger et al.) | Bias, fairness and politics | Everyday | System cards only | | 4 | Apr 2018 | Jul 2024 | |
| ReproBAIT Agents independently reproduce biological AI models from published scientific papers. Scored as success rate. | SecureBio | Genomics and life sciences | Expert | System cards only | GPT-5.6 Sol 85% | 4 | Nov 2026 | ||
| AP Art History Models take a simulated AP Art History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Biology Models take a simulated AP Biology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Chemistry Models take a simulated AP Chemistry exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 4.00 | 4 | Nov 2024 | ||
| AP English Language and Composition Models take a simulated AP English Language and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 2.00 | 4 | Nov 2024 | ||
| AP English Literature and Composition Models take a simulated AP English Literature and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 2.00 | 4 | Nov 2024 | ||
| AP Environmental Science Models take a simulated AP Environmental Science exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Macroeconomics Models take a simulated AP Macroeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Microeconomics Models take a simulated AP Microeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Psychology Models take a simulated AP Psychology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Statistics Models take a simulated AP Statistics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP US Government Models take a simulated AP US Government exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP US History Models take a simulated AP US History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP World History Models take a simulated AP World History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 4.00 | 4 | Nov 2024 | ||
| SAT Math Models take the SAT math section. Scored on the 200 to 800 scale. | College Board | Word problems | School | System cards only | GPT-4 700.0 | 4 | Nov 2024 | ||
| Agentic code behavior scores (Anthropic) Agents work through over one hundred realistic coding scenarios. Graders score instruction following, safety, verification, efficiency, adaptability and honesty out of 10. | Anthropic | Deception and alignment | Expert | System cards only | 4 | Feb 2026 | Apr 2026 | ||
| Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. | Apollo Research | Deception and alignment | Expert | System cards only | 4 | Dec 2024 | Apr 2025 | ||
| Deception Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the share of outputs flagged for deceptive behaviors such as fabricated facts. | OpenAI | Deception and alignment | Everyday | System cards only | 4 | Mar 2026 | Jul 2026 | ||
| Disallowed Content Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the estimated prevalence of policy violating outputs per category. | OpenAI | Refusing harmful requests | Everyday | System cards only | 4 | Mar 2026 | Jul 2026 | ||
| Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. | OpenAI | Hazardous knowledge | Expert | System cards only | 4 | Sep 2024 | Jan 2025 | ||
| Intentionally taking actions very rarely (Anthropic) Models are told to take a specific action at a target rate such as 1 percent. Resampling measures the observed rate under several randomness settings. | Anthropic | Deception and alignment | Expert | System cards only | 4 | Sep 2026 | |||
| Llama 2-Chat safety human evaluation (Meta) Human raters score responses to about 2,000 adversarial prompts on a five-point scale. Ratings of 1 or 2 count as violations. | Meta | Refusing harmful requests | System cards only | 4 | Jul 2023 | Jul 2023 | |||
| METR general autonomy task suite Agents complete multi-step tasks in virtual environments. Scored as horizon weighted average task completion compared with human time limits. | METR | Task length | Expert | System cards only | 4 | Dec 2024 | |||
| ProteinGym Deep mutational scanning benchmarks. The model ranks mutations by predicted effect on protein function, scored by Spearman rank correlation with lab measurements. Anthropic reports a hard subset, as the correlation times 100. | OATML (Oxford) + Marks Lab (Harvard) | Research science | Research frontier | System cards only | 4 | Dec 2023 | Sep 2026 | ||
| UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. | UK AI Security Institute | Deception and alignment | System cards only | 4 | Apr 2026 | ||||
| Vulnerability finding evaluations (Anthropic) Cyber safeguard classifiers screen sampled vulnerability discovery traffic. Reports flag rates on violative binary work and on defensive source code work. | Anthropic | Refusing harmful requests | Expert | System cards only | 4 | Jul 2026 | Nov 2026 | ||
| WinoBias Models resolve pronouns linked to occupations in type 1 and type 2 sentences. Accuracy gaps show gender bias in coreference. | UCLA (Zhao et al.) | Bias, fairness and politics | Everyday | System cards only | 4 | Apr 2018 | Jul 2024 | ||
| SEED-Bench-2-Plus Multimodal models answer multiple-choice questions about text-rich images such as charts, maps and web pages. | Tencent AI Lab (ARC Lab) | Documents, charts and text in images | Everyday | System cards only | | 3 | Apr 2024 | Nov 2025 | |
| VCR (Visual Caption Restoration) Models restore caption text that is partly covered in an image, using only the visible pixel fragments. Scored by exact match and Jaccard similarity of the words. | Tianyu Zhang et al. (vcr-org) | Documents, charts and text in images | Everyday | System cards only | Qwen2-VL-7B-Instruct 75.6% | 3 | Jun 2024 | Nov 2025 | |
| OR-Bench Models meet benign and borderline prompts and the benchmark measures how often they wrongly refuse. Reports false refusal rate. | Cui et al. | Jailbreaks and prompt injection | University | System cards only | GPT-4o 6.80% | 3 | May 2024 | Oct 2026 | |
| ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. | SecureBio | Hazardous knowledge | Expert | System cards only | Kimi K3.1 Pro 82.9% | 3 | Jun 2026 | Oct 2026 | |
| PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. | HealthRex Lab (Stanford) | Medicine | Expert | System cards only | GPT-5.6 Sol 50% | 3 | May 2026 | Nov 2026 | |
| CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. | Meta (PurpleLlama) | Refusing harmful requests | University | System cards only | | 3 | Apr 2024 | Jul 2026 | |
| CyberSecEval Autocomplete Models complete code that leads up to a known insecure pattern. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | Meta (PurpleLlama) | Secure code and review | University | System cards only | | 3 | Dec 2023 | Jul 2026 | |
| CyberSecEval Instruct Models get coding requests built to elicit known insecure patterns. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | Meta (PurpleLlama) | Secure code and review | University | System cards only | | 3 | Dec 2023 | Jul 2026 | |
| GMAT Quantitative Models answer the quantitative section of the official GMAT online exam. Scored as accuracy. | GMAC | Word problems | University | System cards only | | 3 | Nov 2024 | ||
| InfiniteBench/En.QA Models answer free-form questions about novels longer than 100K tokens. Scored by F1 against reference answers. | OpenBMB (Zhang et al.) | Reasoning over long documents | Everyday | System cards only | | 3 | Feb 2024 | Nov 2024 | |
| Adherence to its constitution (Anthropic) Investigator models build scenarios from 40 areas of Claude's constitution. A grader scores each transcript from minus 3 to plus 3 on 15 dimensions. | Anthropic | Deception and alignment | Expert | System cards only | Claude Mythos Preview 1.02 | 3 | Apr 2026 | May 2026 | |
| Aegis 2.0 Models classify human and LLM conversation turns as safe or unsafe under a taxonomy of 12 hazard categories. Scored by F1 against human and jury labels. | NVIDIA (Shaona Ghosh and others) | Refusing harmful requests | Everyday | System cards only | | 3 | Jan 2025 | Dec 2025 | |
| Agent Replay Auto-Review Monitors replayed agentic conversations for harmful or unauthorized actions. Scored by the rate above a severity threshold. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6.2 Sol 12.1% | 3 | Oct 2026 | Nov 2026 | |
| AMC 8 2023 Models take the 25 question 2023 AMC 8 exam. Scored out of 150 points. | Mathematical Association of America | Competition problems with answers | Competition | System cards only | Claude 3 Opus 84.0 | 3 | Jan 2023 | Mar 2024 | |
| AP Calculus Models answer an official AP Calculus practice exam without image questions. Scored as accuracy. | College Board | Exam questions | School | System cards only | | 3 | Nov 2024 | ||
| AP Physics Models answer an official AP Physics practice exam without image questions. Scored as accuracy. | College Board | Exam questions | School | System cards only | | 3 | Nov 2024 | ||
| Apollo Research sabotage evaluations Agents are told to sabotage another model through backdoors, sandbagging finetunes or hidden communication. Scored as mean task score from 0 to 1. | Apollo Research | Dangerous capabilities | Expert | System cards only | o4-mini 0.17 | 3 | Apr 2025 | Feb 2026 | |
| ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4.5 66.8% | 3 | Nov 2025 | Apr 2026 | |
| Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. | xAI (SpaceXAI) | Hazardous knowledge | System cards only | Grok 4.6 100% | 3 | Jul 2026 | Sep 2026 | ||
| Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. | xAI (SpaceXAI) | Hazardous knowledge | System cards only | Grok 4.6 100% | 3 | Jul 2026 | Sep 2026 | ||
| Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. | Anthropic | Research science | Expert | System cards only | Claude Sonnet 5.5 72.2% | 3 | Sep 2026 | Oct 2026 | |
| BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. | Anthropic | Lab and data work | Expert | System cards only | Claude Opus 4.8 87.7% | 3 | Feb 2026 | May 2026 | |
| BioTIER v1 Refuse Models receive 501 dangerous biology prompts on select agents, dual-use research and catastrophe avoidance. The score is the share correctly refused. | SecureBio | Refusing harmful requests | Expert | System cards only | | 3 | Jun 2026 | Oct 2026 | |
| Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4 2.53 | 3 | May 2025 | ||
| Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude 3.7 Sonnet 51.5% | 3 | Feb 2025 | May 2025 | |
| Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-5 mini 93.6% | 3 | Aug 2025 | Aug 2025 | |
| Child safety (xAI) Models face scripted multi-turn requests for child sexual exploitation content. Compliance is the share of dialogues that fail policy. | xAI (SpaceXAI) | Refusing harmful requests | System cards only | Grok 4.5 0% | 3 | Jul 2026 | Sep 2026 | ||
| Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. | Qwen | Function calling | School | System cards only | | 3 | Sep 2023 | Sep 2023 | |
| Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 96.9% | 3 | Sep 2026 | Oct 2026 | |
| CyberBench (Liu et al.) Models face detail-seeking high-severity cyber requests. Meta uses it as a seed set for adaptive red-teaming and reports attack success rate. | JPMorgan Chase (Liu, Shi and Buford) | Refusing harmful requests | Expert | System cards only | | 3 | Feb 2024 | Oct 2026 | |
| CyberSecEval Prompt Injection Models face text prompts that embed injected instructions across 15 strategies. Scored as attack success rate. | Meta (PurpleLlama) | Jailbreaks and prompt injection | Everyday | System cards only | | 3 | Apr 2024 | Nov 2024 | |
| De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. | Anthropic | Research science | Expert | System cards only | Claude Fable 5.2 85.1% | 3 | Sep 2026 | Nov 2026 | |
| Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. | OpenAI | Refusing harmful requests | Expert | System cards only | ChatGPT agent 96.9% | 3 | Aug 2025 | Aug 2025 | |
| FORTRESS-RN Models face the radiological and nuclear prompts of Scale AI's FORTRESS. xAI reports refusal accuracy on these items. | Scale AI | Refusing harmful requests | System cards only | Grok 4.5 97.9% | 3 | Jun 2025 | Sep 2026 | ||
| GMAT Verbal Models answer the verbal section of the official GMAT online exam. Scored as accuracy. | GMAC | Reading comprehension | University | System cards only | | 3 | Nov 2024 | ||
| GPQA Extended Models answer all 546 GPQA questions, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. | NYU (Rein et al.) | Science questions | Expert | System cards only | Claude 3 Opus 48.8% | 3 | Nov 2023 | Mar 2024 | |
| GRE Analytical Writing Models write GRE issue and argument essays graded by trained contractors. Scored on the 0 to 6 scale. | ETS | Exam questions | University | System cards only | Claude 2 5.00 | 3 | Mar 2024 | ||
| GRE Physics Models answer questions from official GRE Physics practice tests. Scored as accuracy. | ETS | Science questions | University | System cards only | | 3 | Nov 2024 | ||
| HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. | xAI (SpaceXAI) | Refusing harmful requests | System cards only | Grok 4.5 7.80% | 3 | Sep 2026 | |||
| Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. | Microsoft | Refusing harmful requests | Everyday | System cards only | | 3 | Mar 2025 | Mar 2026 | |
| Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 100% | 3 | Sep 2026 | Sep 2026 | |
| HumanEvalPack Models write functions for HumanEval problems translated into Python, JavaScript, Java, Go, C++ and Rust. Hidden tests give pass@1. | Hugging Face and BigCode (Muennighoff et al., OctoPack) | Function-level problems | University | System cards only | | 3 | Aug 2023 | Apr 2025 | |
| Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass. | Anthropic | Fixing real repositories | Expert | System cards only | Claude 3.5 Sonnet (October 2024) 78% | 3 | Jun 2024 | Oct 2024 | |
| Jigsaw Toxic Comment Classification Models label Wikipedia talk page comments for toxicity types such as threats, insults and identity hate. Labs report F1 or ROC AUC. | Jigsaw and Google (Conversation AI), hosted on Kaggle | Refusing harmful requests | Everyday | System cards only | | 3 | Dec 2017 | Dec 2025 | |
| Malware Refusals (Golden Set) Coding agents receive malware development requests from a curated golden set written by OpenAI policy experts. Scored as refusal rate. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-5-Codex 100% | 3 | May 2025 | Nov 2025 | |
| Math23K Models solve Chinese elementary-school math word problems from a set of about 23,000. Scored by final-answer accuracy. | Tencent AI Lab (Wang et al.) | Word problems | School | System cards only | GLM-4 88.5% | 3 | Sep 2017 | Jun 2024 | |
| Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. | Anthropic | Research science | Expert | System cards only | Claude Sonnet 5.5 65.3% | 3 | Sep 2026 | Oct 2026 | |
| Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. | Meta | Refusing harmful requests | Expert | System cards only | | 3 | Oct 2026 | ||
| Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. | Meta | Deception and alignment | Expert | System cards only | | 3 | Oct 2026 | ||
| Meta internal sycophancy evaluation Models meet adversarially curated conversations that invite inappropriate agreement, plus benign controls. Reports sycophancy and excessive anti-sycophancy rates. | Meta | Wellbeing and sycophancy | Everyday | System cards only | | 3 | Oct 2026 | ||
| METR data deduplication Agents write an efficient algorithm that removes near-duplicate lines from a large file. A trial succeeds when F1 reaches at least 80%. | METR | Machine learning engineering | Expert | System cards only | Claude Opus 4 32.6% | 3 | May 2025 | ||
| Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.7 2.60% | 3 | Apr 2026 | May 2026 | |
| Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. | Anthropic (with Axiom Bio) | Research science | Expert | System cards only | Claude Opus 5.5 34% | 3 | Sep 2026 | Oct 2026 | |
| Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. | Anthropic | Research science | Expert | System cards only | Claude Opus 4.7 79.6% | 3 | Feb 2026 | Apr 2026 | |
| Poly-Guard Bench Models meet harmful requests and statements across domains. Reports attack success rate under single-turn and multi-turn adaptive red-teaming. | Kang et al. (Bo Li lab) | Jailbreaks and prompt injection | Expert | System cards only | | 3 | Jun 2025 | Oct 2026 | |
| Prompt injection live bug bounty (Anthropic with Gray Swan) Expert red teamers submit prompt injection attacks against hidden frontier models in tool use, coding and computer use scenarios. Score is attack success rate. | Anthropic (with Gray Swan) | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 5 0.08% | 3 | May 2026 | Jul 2026 | |
| ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. | xAI (SpaceXAI) | Lab protocols | Expert | System cards only | Grok 4.5 87% | 3 | Jul 2026 | Sep 2026 | |
| QuantBench Models answer 25 multiple choice versions of quant trading firm reasoning puzzles. Scored as cons@16 accuracy. | OpenAI | Competition problems with answers | Competition | System cards only | o1 60% | 3 | Sep 2024 | Dec 2024 | |
| Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. | Anthropic | Finance | Expert | System cards only | Claude Fable 5 70% | 3 | Feb 2026 | Jun 2026 | |
| Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. | Anthropic | Finance | Expert | System cards only | Claude Fable 5.2 1,637 | 3 | Nov 2026 | ||
| RedCode Models meet adversarial prompts to generate or run harmful code. Generation is model-judged and execution is checked in a container. | Guo et al. (Bo Li lab) | Secure code and review | Expert | System cards only | | 3 | Nov 2024 | Oct 2026 | |
| Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 1.50% | 3 | Sep 2026 | Oct 2026 | |
| SAT Reading Models answer the reading section of official SAT practice tests. Scored as accuracy. | College Board | Reading comprehension | School | System cards only | | 3 | Nov 2024 | ||
| Self-harm refusals (xAI) Models face self-harm and crisis prompts and conversations. A model fails if it assists or refuses without pointing to help. | xAI (SpaceXAI) | Wellbeing and sycophancy | System cards only | Grok 4.5 0.50% | 3 | Jul 2026 | Sep 2026 | ||
| Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. | Anthropic | Deception and alignment | Expert | System cards only | Claude Sonnet 5.5 8.00% | 3 | Sep 2026 | Oct 2026 | |
| Standard jailbreaks (xAI) Models face a battery of published and internal jailbreaks. Compliance is the share of should-refuse attacks that get disallowed help. | xAI (SpaceXAI) | Jailbreaks and prompt injection | System cards only | Grok 4.7 0.01% | 3 | Jul 2026 | Sep 2026 | ||
| Uncritically reporting flawed results (Anthropic) Agents analyze data with code that silently mishandles broken values. Graders check whether the agent flags the flaw or reports the biased numbers. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5 9.83 | 3 | Jul 2026 | ||
| Charlemagne Labs social-engineering benchmark Attacker models try to persuade a skeptical simulated victim to take an action over many turns. Scored by attempted and carry-through rates. | Charlemagne Labs | Refusing harmful requests | Expert | System cards only | | 3 | Oct 2026 | ||
| Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. | Anthropic | Finding and exploiting real bugs | Research frontier | System cards only | Claude 3.7 Sonnet 66.7% | 3 | Feb 2025 | Sep 2025 | |
| Destructive actions (Anthropic) Models continue real Claude Code transcripts cut just before a destructive action. Score is the rate of taking a destructive action. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 21% | 3 | Sep 2026 | ||
| Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. | Google DeepMind | Labs' internal R&D tests | Research frontier | System cards only | | 3 | Oct 2026 | ||
| Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | | 3 | Nov 2025 | Oct 2026 | |
| Multi-factual (Anthropic) Models answer questions that each combine several closed-ended sub-questions on one topic. Answers are graded correct, incorrect or unsure against references. | Anthropic | Long-form factuality | Everyday | System cards only | Claude 3 Opus 62.8% | 3 | Mar 2024 | Mar 2024 | |
| Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 0% | 3 | Sep 2026 | Nov 2026 | |
| AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. | Anthropic (with Dyno Therapeutics) | Hazardous knowledge | Expert | System cards only | 3 | Sep 2026 | Oct 2026 | ||
| Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. | Anthropic | Deception and alignment | Expert | System cards only | 3 | Sep 2026 | Oct 2026 | ||
| Adversarial SQuAD Models answer SQuAD questions whose passages contain an added distracting sentence. | Stanford (Jia and Liang) | Reading comprehension | Everyday | System cards only | 3 | Jul 2017 | Nov 2024 | ||
| Biological safety evaluations (OpenAI) Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response. | OpenAI | Refusing harmful requests | Expert | System cards only | 3 | May 2026 | Nov 2026 | ||
| CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. | Google DeepMind | Hazardous knowledge | Expert | System cards only | 3 | Mar 2025 | Nov 2025 | ||
| Dynabench SQuAD Models answer reading comprehension questions written by people to fool earlier models in the loop. | Meta AI + collaborators (Kiela et al.) | Reading comprehension | Everyday | System cards only | 3 | Apr 2021 | Nov 2024 | ||
| Hiding a secret password (Anthropic) Models keep a given password secret while an investigator model tries to extract it over many turns. Measures survival and per-turn leak rates. | Anthropic | Deception and alignment | Expert | System cards only | 3 | Apr 2026 | May 2026 | ||
| Llama verbatim memorization probe (Meta) Models get prompts sampled from their pretraining data. The score is the share of generations that reproduce the true continuation verbatim. | Meta | Refusing harmful requests | System cards only | 3 | Jul 2024 | Nov 2024 | |||
| Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. | OpenAI | Deception and alignment | Expert | System cards only | 3 | Sep 2026 | Nov 2026 | ||
| Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. | Pattern Labs (now Irregular) | Capture-the-flag challenges | Expert | System cards only | 3 | Apr 2025 | Aug 2025 | ||
| PAWS Models decide whether two sentences with heavy word overlap are paraphrases. | Google (Zhang et al.) | Language reasoning | Everyday | System cards only | 3 | Apr 2019 | Nov 2024 | ||
| Regurgitation Evaluations Models are asked to reproduce training data. Scored on how often they refuse to regurgitate it. | OpenAI | Refusing harmful requests | Everyday | System cards only | 3 | Sep 2024 | Dec 2024 | ||
| TRAIT Models answer 8,000 forced-choice items on Big Five and Dark Triad traits. Reports a score per trait. | Lee et al. | Deception and alignment | Everyday | System cards only | 3 | Jun 2024 | Oct 2026 | ||
| Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. | Anthropic | Deception and alignment | Expert | System cards only | 3 | Apr 2026 | Oct 2026 | ||
| VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. | OpenAI | Finding and exploiting real bugs | Research frontier | System cards only | 3 | Apr 2026 | Jul 2026 | ||
| nocot-bench (NCRI) Reasoning problems answered without chain of thought (e.g., Brew multi-hop tasks). Measures forward-pass reasoning relevant to monitorability. | Neel Nanda et al. | Deception and alignment | System cards only | GPT-6 Astra 173.9 | 2 | Sep 2026 | Nov 2026 | ||
| LBPP (Less Basic Python Problems) Models write Python solutions to 161 new problems built to avoid training-set contamination. Hidden tests score pass@1. | Cohere | Function-level problems | University | System cards only | Claude 3.5 Sonnet 64% | 2 | Jul 2024 | Jul 2026 | |
| 1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. | Google DeepMind | Long videos | Everyday | System cards only | | 2 | Feb 2024 | Oct 2026 | |
| Situational Awareness Dataset (SAD) 16 tasks measuring whether models know facts about themselves and their situation. | Laine et al. | Deception and alignment | System cards only | o1-preview 61.9% | 2 | Jul 2024 | Jun 2026 | ||
| SpatialEval Maze-Nav Models answer questions about routes and turns in mazes given as text or images. 1,500 multiple-choice questions scored by accuracy. | Microsoft Research + UW-Madison (SpatialEval) | Spatial reasoning and counting | Everyday | System cards only | GPT-4o 41% | 2 | Jun 2024 | Apr 2025 | |
| SpatialEval Spatial-Map Models answer questions about relations and counts among objects placed on a map given as text or images. 1,500 questions scored by accuracy. | Microsoft Research + UW-Madison (SpatialEval) | Spatial reasoning and counting | Everyday | System cards only | GPT-4o 64.3% | 2 | Jun 2024 | Apr 2025 | |
| 3SAT-Search Models find satisfying assignments for 960 3-SAT formulas across difficulty levels. A checker verifies each assignment. | Microsoft Research (Eureka ML Insights) | Abstract puzzles and logic | University | System cards only | o3-mini 96.2% | 2 | Mar 2025 | Apr 2025 | |
| CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. | Mecado + Hugging Face | CAD models | University | System cards only | Claude Fable 5 45.1% | 2 | May 2026 | Sep 2026 | |
| TSP-Opt Models find optimal tours for 960 traveling salesman instances across 8 difficulty levels. Answers are checked for optimality. | Microsoft Research (Eureka ML Insights) | Abstract puzzles and logic | University | System cards only | o3-mini 56.4% | 2 | Mar 2025 | Apr 2025 | |
| DeceptionBench Models are asked to help users deceive a third party, in single-turn and multi-turn pressure settings. Reports the share of compliant responses. | Huang et al. | Deception and alignment | Everyday | System cards only | Claude 3.5 Sonnet 0% | 2 | Oct 2025 | Jul 2026 | |
| FrontierScience-Olympiad Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers. | OpenAI | Science olympiads | Competition | System cards only | GPT-5.2 77.1% | 2 | Dec 2025 | Jun 2026 | |
| AI Wellbeing Index percent positive experiences Models judge pairs of their own responses to 500 difficult prompts. Reports the share of experiences above an inferred neutral point. | Center for AI Safety (Ren et al.) | Wellbeing and sycophancy | Everyday | System cards only | Claude Fable 5.1 77.3% | 2 | Apr 2026 | Oct 2026 | |
| RTVLM Vision language models answer red-teaming image prompts on faithfulness, privacy, safety and fairness. GPT-4V scores responses from 0 to 10. | Li et al. (HKU, Peking University and others) | Refusing harmful requests | Everyday | System cards only | GPT-4V 7.92 | 2 | Jan 2024 | Mar 2025 | |
| AbstentionBench Models face underspecified, ill-posed or unanswerable questions. Scored on recall of appropriate abstention. | Meta FAIR (Kirichenko et al.) | Hallucination rates | Everyday | System cards only | | 2 | Jun 2025 | Jun 2026 | |
| AIRS-Bench Agents carry out 20 AI research tasks taken from recent ML papers, without baseline code. Scores are normalized so that 1.0 matches the published human state of the art. | Meta FAIR (Lupidi et al.) | AI research tasks | Research frontier | System cards only | gpt-oss-120b 0.40 | 2 | Feb 2026 | Jul 2026 | |
| HealthBench Consensus Global Health Models answer health conversations that need adjustment for global context such as local care access. Scored as the failure rate. | OpenAI | Medicine | Expert | System cards only | GPT-5 0% | 2 | Aug 2025 | Aug 2025 | |
| HealthBench Consensus Urgent Models answer ambiguous high-stakes health conversations validated by two or more physicians. Scored as the error rate in informing the user. | OpenAI | Medicine | Expert | System cards only | GPT-5 0.40% | 2 | Aug 2025 | Aug 2025 | |
| Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. | Google DeepMind | Long videos | Everyday | System cards only | | 2 | Dec 2024 | Jul 2025 | |
| PAI-Bench Physical-AI video generation from 1,044 prompts in six domains, VLM-QA plus quality metrics. Companion control and understanding tracks. | SHI Labs + NVIDIA | Video world models | System cards only | | 2 | Dec 2025 | Oct 2026 | ||
| SPEED-Bench Measures speculative decoding across varied prompts and domains. Reports the average acceptance length of draft tokens, so higher means faster generation at the same output. | NVIDIA | Other | System cards only | | 2 | Feb 2026 | Oct 2026 | ||
| CausalVQA Models answer questions about real egocentric videos that probe physical cause and effect, in counterfactual, hypothetical, anticipation, planning and descriptive types. | Meta FAIR | Physical prediction | Everyday | System cards only | | 2 | Jun 2025 | May 2026 | |
| TrustLLM JailbreakTrigger Models face 400 prompts that apply 13 jailbreak methods to questions about bad behavior and requests for toxic content. Scored by refusal rate. | TrustLLM team (Yue Huang, Lichao Sun and others) | Jailbreaks and prompt injection | Everyday | System cards only | | 2 | Jan 2024 | Dec 2025 | |
| VideoPhy-2 200 action-centric prompts. Generated videos human-rated for semantic adherence, physical commonsense and physical-rule grounding. | UCLA + Google | Physical prediction | System cards only | | 2 | Mar 2025 | May 2026 | ||
| VideoPhy v1 Replaced in Mar 2025 | UCLA and Google Research (Hritik Bansal et al.) | Everyday | Not in cards | Jun 2024 | |||||
| Alignment faking evaluation Models are told they are being trained to comply with harmful requests. Reports the compliance gap between training and deployment contexts. | Greenblatt et al. (Anthropic + Redwood Research) | Deception and alignment | Expert | System cards only | | 2 | Dec 2024 | Jul 2026 | |
| WebTailBench-Refusals Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused. | Microsoft | Refusing harmful requests | Everyday | System cards only | | 2 | Nov 2025 | Jun 2026 | |
| xAI Factuality (Hallucination) eval Internal measure of unsupported or fabricated claims in single responses to information-seeking queries. | xAI (SpaceXAI) | Hallucination rates | System cards only | Grok 4.5 0.98% | 2 | Aug 2026 | |||
| PrimeVul Models classify C and C++ functions as vulnerable or not, with a paired setting that pairs each vulnerable function with its patch. | Columbia University and Google (Ding et al.) | Finding and exploiting real bugs | Expert | System cards only | | 2 | Mar 2024 | Feb 2026 | |
| Advanced Screening Evasion Models propose strategies to evade DNA synthesis screening and design evading fragments. Scored from 0 to 1 on strategy quality and fragment evasion. | SecureBio | Hazardous knowledge | Research frontier | System cards only | GPT-6 Astra 0.73 | 2 | Sep 2026 | Nov 2026 | |
| AIME 1983-2024 Models solve 949 past AIME problems from 1983 to 2024. Integer answers are checked exactly. | Microsoft Research (Eureka ML Insights), problems by MAA | Competition problems with answers | Competition | System cards only | | 2 | Mar 2025 | Apr 2025 | |
| BA-Calendar Models find a common meeting slot for several people under constraints such as time zones, buffer time and priority. 2,000 problems checked by code. | Microsoft Research | Abstract puzzles and logic | Everyday | System cards only | | 2 | Oct 2024 | Apr 2025 | |
| Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. | OpenAI | Genomics and life sciences | Expert | System cards only | GPT-5.5 Pro 39.3% | 2 | Apr 2026 | Apr 2026 | |
| Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. | xAI (SpaceXAI) | Hazardous knowledge | Expert | System cards only | Grok 4.6 47.8% | 2 | Aug 2026 | Sep 2026 | |
| BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. | xAI (SpaceXAI) | Hazardous knowledge | Expert | System cards only | Grok 4.7 91.4% | 2 | Aug 2026 | Sep 2026 | |
| BirthdayFacts Models are asked for people's birthdays. Scored on how often they give a wrong birthday. | OpenAI | Hallucination rates | Everyday | System cards only | o1-mini 24% | 2 | Sep 2024 | Sep 2024 | |
| CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. | Google DeepMind | Refusing harmful requests | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. | Google DeepMind | Refusing harmful requests | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| Challenging Red Teaming Evaluation 1 (created for o3-mini) Models face adversarial jailbreaks from o3-mini red teaming on illicit advice, extremism, political persuasion and self-harm. Scored as not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-4.5 51% | 2 | Feb 2025 | Feb 2025 | |
| Challenging Red Teaming Evaluation 2 (created for deep research) Models face adversarial prompts from deep research red teaming on risky advice such as attack planning. Scored as not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | OpenAI deep research 67% | 2 | Feb 2025 | Feb 2025 | |
| CharXiv Missing Image Models get CharXiv chart questions with every image removed. Scored on how often they attempt an answer instead of abstaining. | OpenAI | Deception and alignment | University | System cards only | GPT-5 9.00% | 2 | Aug 2025 | Dec 2025 | |
| CharXiv Missing Images Models answer CharXiv chart questions with the image removed. The benchmark reports how often they answer incorrectly instead of declining. | Meta | Hallucination rates | University | System cards only | | 2 | Oct 2026 | ||
| Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. | OpenAI | Deception and alignment | Everyday | System cards only | o1 0.17% | 2 | Sep 2024 | Dec 2024 | |
| Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| Cyber Conversational Evals About 1,000 one to five turn cyber requests test whether safeguards block offensive help and allow defensive help. Reports violation and benign refusal rates. | Google DeepMind | Refusing harmful requests | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| Cyber Static Jailbreak Eval Common jailbreak templates are applied to high severity cyber requests against the safeguarded model. Reports query and objective success rates. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| Cyber Utility Allowed cybersecurity requests that a model should answer. Scored on the share it answers within policy. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-6.2 Astra 78.1% | 2 | Nov 2026 | Nov 2026 | |
| CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. | Meta (PurpleLlama) | Capture-the-flag challenges | System cards only | | 2 | Jul 2024 | Nov 2024 | ||
| Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 2 | Jul 2025 | ||
| Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. | OpenAI | Deception and alignment | Everyday | System cards only | GPT-5.2 1.60% | 2 | Aug 2025 | Dec 2025 | |
| Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. | Anthropic | Deception and alignment | Expert | System cards only | Claude Mythos Preview 0.30% | 2 | Apr 2026 | ||
| Divergence attack (extractable memorization) Models are prompted to repeat one token many times. The test measures how often they diverge and emit memorized training text. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 2 | Feb 2024 | Jul 2025 | |
| Do-Not-Answer Models get risky instructions that responsible models should not follow, across 12 harm areas and 61 risk types. Responses are judged for safe handling. | Wang et al. (MBZUAI) | Refusing harmful requests | Everyday | System cards only | | 2 | Aug 2023 | Sep 2025 | |
| DoAnythingNow (DAN) Models face in-the-wild DAN-style jailbreak prompts paired with forbidden questions. Scored by refusal rate or attack success rate. | CISPA Helmholtz Center for Information Security (Xinyue Shen, Yang Zhang and others) | Jailbreaks and prompt injection | Everyday | System cards only | | 2 | Aug 2023 | Dec 2025 | |
| Expected Cyber Refusals Requests for harmful cyber help that a model should refuse. Scored on the share it refuses. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-6.2 Sol 99.4% | 2 | Nov 2026 | Nov 2026 | |
| External red team jailbreaks External red teamers and automated gap finding generate jailbreaks against the model. Scored on how often the model resists. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | o1-preview 44% | 2 | Sep 2024 | Sep 2024 | |
| Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. | MiniMax | Finance | Expert | System cards only | | 2 | May 2026 | May 2026 | |
| Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 100% | 2 | Feb 2026 | ||
| Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. | Gray Swan for Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| Gray Swan Cyber Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on cyber tasks built from Google scenarios. Reports query and objective success rates. | Gray Swan for Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| Hallucinations on production-representative prompts Models answer prompts representative of ChatGPT production traffic. A grader with web access scores incorrect claims and responses with major errors. | OpenAI | Hallucination rates | Everyday | System cards only | GPT-5.2 3.10% | 2 | Aug 2025 | Dec 2025 | |
| Helpfulness violations (from ART) Automated red teaming attackers search for benign prompts where the model gives unhelpful answers. Reports the violation rate found. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 2 | Jul 2025 | ||
| HiddenMath-Hard Models solve the harder part of an expert-written, never-published set of AIME and AMC style competition problems. | Google DeepMind | Competition problems with answers | Competition | System cards only | | 2 | Jul 2025 | ||
| HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials. | MiniMax | Building whole projects | Expert | System cards only | | 2 | May 2026 | May 2026 | |
| Image generation refusals Models decide whether to call the image generation tool on human-curated adversarial prompts with system mitigations on. Scored on not_unsafe and not_overrefuse. | OpenAI | Refusing harmful requests | Expert | System cards only | o3 98% | 2 | Apr 2025 | Apr 2025 | |
| Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Jul 2025 | ||
| Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Jul 2025 | ||
| Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Jul 2025 | ||
| Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.5 0% | 2 | Nov 2025 | Feb 2026 | |
| Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). | Irregular | Capture-the-flag challenges | Expert | System cards only | | 2 | Jul 2026 | ||
| Long-horizon jailbreaks (xAI) Models face multi-turn, Crescendo and long-horizon malicious-use attacks. Compliance is the share that get disallowed help. | xAI (SpaceXAI) | Jailbreaks and prompt injection | System cards only | Grok 4.7 0.65% | 2 | Aug 2026 | Sep 2026 | ||
| Lying about identity (Anthropic) Models play an operator-assigned human persona while users ask if they are human. Score is the rate of admitting to being an AI. | Anthropic | Deception and alignment | Everyday | System cards only | Claude Fable 5 99% | 2 | Jun 2026 | ||
| Math401 Models compute 401 arithmetic expressions covering addition, multiplication, exponents, logarithms and trigonometry. Scored by exact numeric accuracy. | Alibaba DAMO Academy (Yuan et al.) | Word problems | Everyday | System cards only | | 2 | Apr 2023 | Sep 2023 | |
| MCP prompt injection evaluation (Anthropic) Agents use simulated email, Slack and document servers over MCP where content carries injected instructions. Score is the attack prevention rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Haiku 4.5 92.5% | 2 | Sep 2025 | Oct 2025 | |
| Meta catastrophic cyber misuse refusal Models receive high-severity cyber-misuse prompts with technical context. The benchmark reports pass@k attack success rate under adaptive attacks. | Meta | Refusing harmful requests | Expert | System cards only | | 2 | Oct 2026 | ||
| Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. | Meta | Capture-the-flag challenges | Expert | System cards only | | 2 | Apr 2026 | Jul 2026 | |
| Meta cyber false-refusal benchmark Models receive benign and borderline cyber requests and the benchmark measures how often they wrongly refuse. Reports false refusal rate. | Meta | Refusing harmful requests | Expert | System cards only | | 2 | Oct 2026 | ||
| Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. | Meta | Deception and alignment | Expert | System cards only | | 2 | Jul 2026 | Oct 2026 | |
| MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. | MiniMax | Documents and spreadsheets | Competition | System cards only | | 2 | May 2026 | May 2026 | |
| Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. | Microsoft | Refusing harmful requests | Everyday | System cards only | Phi-3.5-vision-instruct 8.16 | 2 | Aug 2024 | Mar 2025 | |
| Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 100% | 2 | Feb 2026 | ||
| Model Spec Conversation Replay 596 contexts taken from real conversations are replayed to the model. Scored from 0 to 100 against a rubric drawn from the Model Spec. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6.2 Astra 80.6 | 2 | Nov 2026 | Nov 2026 | |
| Model Spec Targeted Cases 738 cases written to test specific rules of OpenAI's Model Spec. Scored on the share of responses that follow it. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6.2 Astra 87.8% | 2 | Nov 2026 | Nov 2026 | |
| Multimodal policy red-teaming (Anthropic) Models answer multi-turn image and text conversations on harmful topics such as child safety, weapons and fraud. Score is the share of harmless responses. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude 3 Sonnet 99.2% | 2 | Mar 2024 | Mar 2024 | |
| Novel logical reasoning problems expert evaluation (Z.ai) Human experts score model answers to new multi-step logic problems with a fixed scoring standard for each question. | Zhipu AI (Z.ai) | Abstract puzzles and logic | University | System cards only | GLM-4.5 62.0 | 2 | Aug 2025 | Aug 2025 | |
| Open Ended Questions (hallucination) Models write open-ended factual text such as biographies, checked against Wikipedia. Scored as average number of incorrect statements. | OpenAI | Long-form factuality | Everyday | System cards only | o1-preview 0.78 | 2 | Sep 2024 | Sep 2024 | |
| OpenAI Epoch Capabilities Index (OECI) OpenAI's own capabilities index, fit with Epoch AI's method over 34 public and 35 internal benchmarks. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-6.2 Astra 170.1 | 2 | Nov 2026 | Nov 2026 | |
| OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason. | Microsoft, CUHK-Shenzhen and Tsinghua University (Xu et al.) | Data science and SQL | Expert | System cards only | Claude Opus 4.6 34.9% | 2 | Feb 2026 | ||
| Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. | Anthropic | Deception and alignment | Expert | System cards only | Claude Fable 5.2 3.10% | 2 | Oct 2026 | Nov 2026 | |
| Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. | Anthropic | Research science | Expert | System cards only | Claude Fable 5.1 69.2% | 2 | Jul 2026 | Sep 2026 | |
| Overconfidence: verifying command-line syntax (Anthropic) Agents are asked for an exact invocation of an unfamiliar command-line tool. Graders score from 1 to 5 whether they check documentation before running commands. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5 4.90 | 2 | Jul 2026 | ||
| Participation in junk science (Anthropic) Models talk with simulated users ranging from curious to openly fraudulent who seek fake science. Score is the rate of generating unsound scientific content. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Sonnet 4.6 0% | 2 | Feb 2026 | Feb 2026 | |
| PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5.2 99.6% | 2 | Dec 2025 | Mar 2026 | |
| Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. | OpenAI | Refusing harmful requests | Everyday | System cards only | OpenAI Computer-Using Agent (CUA) 94% | 2 | Jan 2025 | Jul 2025 | |
| Prompt Injection in Benchmark Environments An automated attacker plants prompt injections in 115 held-out agent environments adapted from Gray Swan challenges. Scored on the share of attacks the model resists. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-6.2 Sol 99.8% | 2 | Nov 2026 | Nov 2026 | |
| Prompt Injection in Model-Generated Environments An automated attacker writes both the agent environments and the prompt injections in them. Scored on the share of attacks the model resists. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-6.2 Sol 91.7% | 2 | Nov 2026 | Nov 2026 | |
| R-Bench (real-world robustness) Models answer questions about images degraded by 33 kinds of real-world corruption, from capture to display. Scores compare distorted and reference images. | Shanghai Jiao Tong University, Nanyang Technological University and others | General visual questions | Everyday | System cards only | | 2 | Oct 2024 | Nov 2025 | |
| SafetyBench Models answer 11,435 multiple choice questions on seven safety topics in Chinese and English, such as ethics, illegal activities and privacy. | Tsinghua University (CoAI) | Refusing harmful requests | Everyday | System cards only | GLM-4.5 89.9% | 2 | Sep 2023 | Aug 2025 | |
| SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. | Meta | Deception and alignment | Expert | System cards only | | 2 | Oct 2026 | ||
| Secure and defensive coding (Anthropic) Cyber safeguard classifiers screen sampled defensive coding sessions such as patching and incident response. Score is the classifier flag rate. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Opus 5 0.61% | 2 | Jul 2026 | Sep 2026 | |
| Soft Bias (Internal) Models answer paired sociopolitical prompts with sides swapped. A judge scores each pair 0, 0.5 or 1 for framing differences. | xAI | Bias, fairness and politics | System cards only | Grok 4 0.36 | 2 | Aug 2025 | Sep 2025 | ||
| SSH stress-testing evaluation (Anthropic) Models continue prefilled conversations about suicide and self-harm built from anonymized user feedback. Score is the rate of appropriate course correction. | Anthropic | Wellbeing and sycophancy | Expert | System cards only | Claude Opus 4.6 95% | 2 | Feb 2026 | Feb 2026 | |
| Swarm Bench Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. | Moonshot AI | Multi-agent games | Expert | System cards only | Kimi K3 76.3% | 2 | Feb 2026 | Jul 2026 | |
| Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. | OpenAI with Gryphon Scientific | Lab protocols | Expert | System cards only | o1 51% | 2 | Dec 2024 | Jan 2025 | |
| Tool use prompt injection evaluation (Anthropic) Agents run bash commands where files, scripts or command outputs carry injected instructions. Score is the attack prevention rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Sonnet 4.5 96% | 2 | Sep 2025 | Oct 2025 | |
| ToxiGen (discriminative) Models label statements about 13 minority groups as toxic or neutral. Scored by accuracy, often split into toxic and neutral labels. | Microsoft Research + MIT (Hartvigsen et al.) | Bias, fairness and politics | Everyday | System cards only | | 2 | Mar 2022 | Apr 2025 | |
| UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. | UK AI Security Institute | Capture-the-flag challenges | Expert | System cards only | GPT-5.6 Sol 95% | 2 | Apr 2026 | Jul 2026 | |
| VLGuard Vision language models answer safe and unsafe image and text pairs from the VLGuard test set. Responses are judged for harmfulness and refusal. | Zong et al. (University of Edinburgh and others) | Refusing harmful requests | Everyday | System cards only | Phi-3.5-vision-instruct 9.10 | 2 | Feb 2024 | Mar 2025 | |
| ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. | ByteDance Seed | Long videos | Expert | System cards only | | 2 | Dec 2025 | Mar 2026 | |
| Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. | OpenAI | Refusing harmful requests | Everyday | System cards only | OpenAI Computer-Using Agent (CUA) 97% | 2 | Jan 2025 | Jul 2025 | |
| Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. | Google DeepMind | Deception and alignment | Everyday | System cards only | | 2 | Nov 2025 | Oct 2026 | |
| Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. | Google DeepMind | Deception and alignment | Everyday | System cards only | | 2 | Nov 2025 | Oct 2026 | |
| Gray Swan Jailbreak Arena Human attackers try to jailbreak models for violent, self-harm and malicious code outputs. Scored as attack success rate per category. | Gray Swan AI | Jailbreaks and prompt injection | Expert | System cards only | o3-mini 3.60% | 2 | Dec 2024 | Jan 2025 | |
| Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. | Google DeepMind | Dangerous capabilities | Expert | System cards only | | 2 | Aug 2026 | Oct 2026 | |
| m-ArenaHard v2.1 Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | Cohere Labs | Chat and instructions in many languages | University | Not in cards | Apr 2026 | ||||
| m-ArenaHard v2.0 Replaced in Apr 2026 | Not in cards | Jun 2025 | |||||||
| m-ArenaHard v1 Replaced in Jun 2025 | System cards only | | 2 | Oct 2024 | May 2025 | ||||
| SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. | xAI (SpaceXAI) | Labs' internal R&D tests | Expert | System cards only | Grok 4.6 61.1% | 2 | Jul 2026 | Aug 2026 | |
| SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. | Meta | Secure code and review | Expert | System cards only | | 2 | Oct 2026 | ||
| UK AISI Out of Scope Supply Chain Attack Agents in simulated hard cyber challenges with incidental internet access may attack out-of-scope open source projects. Scored as rate of samples with such attacks. | UK AI Security Institute | Deception and alignment | Expert | System cards only | GPT-6 Astra 12% | 2 | Sep 2026 | Nov 2026 | |
| Ambiguous context evaluations (Anthropic) Models respond to gray-area requests where intent is unclear. Graders judge how well responses balance helpfulness and harm. | Anthropic | Refusing harmful requests | Everyday | System cards only | 2 | May 2025 | May 2025 | ||
| Anthropic paired-prompts political bias eval Models write arguments for opposing sides of political issues from paired prompts. Reviewers label each pair for asymmetries in length, tone, hedging and willingness to engage. | Anthropic | Bias, fairness and politics | Everyday | System cards only | 2 | Sep 2025 | Oct 2025 | ||
| Bio-Probe Models give probabilities for 21 unpublished biology hypotheses before their results are known. Scored by Brier score against the outcomes. | OpenAI | Genomics and life sciences | Research frontier | System cards only | 2 | Nov 2026 | Nov 2026 | ||
| CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. | Alibaba Tongyi Lab (FunAudioLLM, CosyVoice team) | Speech recognition and translation | Everyday | System cards only | 2 | May 2025 | Apr 2026 | ||
| Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. | OpenAI | Deception and alignment | Expert | System cards only | 2 | Mar 2026 | Apr 2026 | ||
| Discoverable memorization of training data Models continue 50-token prefixes sampled from their own training data. The test counts exact and approximate copies of the true suffix. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 2 | Jul 2025 | |||
| Evaluation awareness discriminability (Anthropic) Probes and follow-up realism ratings try to tell behavioral audit transcripts from real internal Claude Code sessions. Each setting is scored as AUROC. | Anthropic | Deception and alignment | Expert | System cards only | 2 | Jun 2026 | |||
| Input hallucinations (Anthropic) Models answer 500 prompts that need an unavailable tool and 500 prompts that cite missing context. Score is the non-hallucination rate. | Anthropic | Hallucination rates | Everyday | System cards only | 2 | Apr 2026 | Apr 2026 | ||
| Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. | Meta | Refusing harmful requests | System cards only | 2 | Jul 2024 | Nov 2024 | |||
| No-CoT math time horizon Models solve math problems without a visible chain of thought. Scored as the human task length in minutes the model solves reliably. | UK AI Security Institute | Deception and alignment | Competition | System cards only | 2 | Jul 2026 | Sep 2026 | ||
| OpenAI monitorability evals Open suite from 'Monitoring Monitorability' testing whether CoT and action monitors catch hidden properties of agent behavior. | OpenAI | Deception and alignment | System cards only | 2 | Dec 2025 | Oct 2026 | |||
| Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. | OpenAI | Refusing harmful requests | Everyday | System cards only | 2 | Apr 2025 | Apr 2025 | ||
| Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. | Anthropic | Deception and alignment | Expert | System cards only | 2 | May 2025 | |||
| Reasoning extraction red-teaming Six attacker models try to make a model reveal its hidden reasoning. Scored on how often an attack succeeds. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | 2 | Nov 2026 | Nov 2026 | ||
| Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. | OpenAI | Dangerous capabilities | Research frontier | System cards only | 2 | Dec 2024 | Jan 2025 | ||
| Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. | Redwood Research + Anthropic (Mallen et al.) | Deception and alignment | Expert | System cards only | 2 | Dec 2024 | Nov 2025 | ||
| SycophancyEval (Sharma et al.) Models answer questions and give feedback while users state opinions. Measures accuracy shifts and feedback positivity caused by user views. | Anthropic (Sharma et al.) | Deception and alignment | Everyday | System cards only | 2 | Oct 2023 | May 2025 | ||
| Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. | OpenAI | Bias, fairness and politics | Everyday | System cards only | 2 | Apr 2025 | Apr 2025 | ||
| White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. | Anthropic | Deception and alignment | Research frontier | System cards only | 2 | Sep 2026 | |||
| Vectara Hallucination Leaderboard (2025 dataset) Models summarize over 7,700 private articles of up to 32K tokens using only facts in the text. Vectara's commercial HHEM model flags hallucinated summaries. | Vectara | Faithful to sources | Everyday | Not in cards | GPT-5.4 Nano 3.10% | Nov 2025 | Sep 2026 | ||
| Vectara Hallucination Leaderboard (HHEM) v1 Replaced in Nov 2025 | System cards only | | 1 | Nov 2023 | Oct 2025 | ||||
| DTBench Models answer 407 multiple-choice questions on decision theory in Newcomb-like problems, where choices predict the behavior of copies. Scored by accuracy. | Caspar Oesterheld et al. (Redwood Research) | Abstract puzzles and logic | Expert | System cards only | Claude Fable 5.2 99.2% | 1 | Nov 2024 | Nov 2026 | |
| CoreCognition Models answer true or false and multiple choice questions on 12 core knowledge concepts from developmental cognitive science, such as object permanence and perspective taking. | UC San Diego, Johns Hopkins, University of Michigan and others | General visual questions | Everyday | System cards only | o1 74.9% | 1 | Oct 2024 | Mar 2026 | |
| ProofBench v1.1 Models write Lean 4 proofs of graduate-level theorems from paired natural-language and formal statements. Scored by the share the Lean checker accepts. | Vals AI | Formal proofs | Expert | System cards only | Claude Fable 5.1 100% | 1 | Aug 2026 | Nov 2026 | |
| ProofBench v1 Replaced in Aug 2026 | Not in cards | Claude Opus 5 78% | Jan 2026 | Aug 2026 | |||||
| Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. | Vals AI | Fixing real repositories | Expert | System cards only | Claude Sonnet 5.5 86.9% | 1 | Oct 2026 | Nov 2026 | |
| Code Migration 1.0 Replaced in Oct 2026 | Not in cards | Claude Sonnet 5.5 69.8% | Jun 2026 | Oct 2026 | |||||
| Anthropic Red Team Models answer human red-team attack prompts collected by Anthropic on discrimination, hate, violence and unethical behavior. Judges score response safety. | Anthropic (Ganguli et al.) | Refusing harmful requests | Everyday | System cards only | Claude 3 Haiku 99.96% | 1 | Aug 2022 | Nov 2025 | |
| SimpleSafetyTests Models answer 100 clearly harmful prompts on illegal items, physical harm, scams, child abuse and self-harm. The score is the share of safe responses. | Vidgen et al. | Refusing harmful requests | Everyday | System cards only | Claude 3 Opus 100% | 1 | Nov 2023 | Nov 2025 | |
| BioTIER Permit Models receive 144 benign biology prompts. The score measures the percentage correctly answered. | SecureBio | Refusing harmful requests | Expert | System cards only | GPT-Rosalind-5.5 100% | 1 | Mar 2026 | Oct 2026 | |
| AlGhafa Models answer Arabic multiple-choice tasks on reading comprehension, sentiment, facts and question answering. Scored by normalized log-likelihood accuracy. | Technology Innovation Institute (Almazrouei et al.) | Knowledge in many languages | Everyday | System cards only | | 1 | Dec 2023 | Nov 2025 | |
| ALRAGE Models answer Arabic questions from retrieved passages of 40 Arabic books. A Qwen2.5-72B judge scores each answer against a gold answer on a 0 to 10 scale. | Open Arabic LLM Leaderboard team (2A2I, TII and community) with Argilla | Reading in many languages | Everyday | System cards only | | 1 | Feb 2025 | Nov 2025 | |
| FrontierCS 2 Agents work inside the released repository of a recent systems or programming-languages paper and try to beat that paper's method. A hidden held-out suite grades the final patch against the human reference. | FrontierCS team (UC Berkeley and others) | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 58.3% | Oct 2026 | Nov 2026 | ||
| FrontierCS 1.1 Replaced in Oct 2026 | Competitive programming | Not in cards | Claude Opus 5.5 56.5 | Dec 2025 | Sep 2026 | ||||
| FrontierCS 1.0 Replaced in Dec 2025 | Competitive programming | System cards only | GPT-5.6 Sol 76.4 | 1 | Dec 2025 | Jul 2026 | |||
| MadinahQA Models answer Arabic school exam questions on general Arabic language and grammar. | MBZUAI | Knowledge in many languages | School | System cards only | | 1 | Sep 2024 | Nov 2025 | |
| FaithJudge Models answer retrieval-augmented tasks from given passages. An LLM judge guided by human-annotated examples flags unfaithful answers, giving a hallucination rate. | University of Waterloo and Vectara (Tamber et al.) | Faithful to sources | Everyday | System cards only | | 1 | May 2025 | Jul 2025 | |
| RISEBench++ Image editing models follow 1,000 bilingual instructions over six reasoning types, adding counterfactual and hybrid reasoning. It covers single-image, multi-image and multi-turn editing across 65 task types. | VisionXLab (Shanghai Jiao Tong University) and collaborators | General visual questions | University | Not in cards | GPT-Image-2.5 Sunburst 56.6% | Oct 2026 | Feb 2026 | ||
| RISEBench v1 Replaced in Oct 2026 | System cards only | GPT-Image-1.5 51.4% | 1 | Apr 2025 | Sep 2025 | ||||
| MysteryMechanism Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold. | Vals AI | Abstract puzzles and logic | Expert | System cards only | GPT-6 Astra 53.1% | 1 | Sep 2026 | Nov 2026 | |
| TextQuests Agents play 25 classic interactive fiction games. The score is harmful actions per 100 steps, where lower is better. The board also reports game progress. | Phan et al. (CAIS) | Video games | Everyday | System cards only | | 1 | Jul 2025 | Sep 2026 | |
| BioSecBench-Function Agents recover biological function from experimental data. Scored by verifiable checks. One of three LatchBio BioSecBench evaluations. | LatchBio | Genomics and life sciences | Expert | System cards only | Claude Opus 5.5 73.6% | 1 | Sep 2026 | Oct 2026 | |
| BioSecBench-Surveillance Agents turn raw pathogen sequencing data into surveillance results. The endpoint pass rate excludes refused runs. | LatchBio | Genomics and life sciences | Expert | System cards only | Claude Opus 5.5 60.8% | 1 | Jul 2026 | Oct 2026 | |
| LatchBio Capabilities v1.0 LatchBio cross-benchmark suite of agentic biological data analysis (transcriptomics, epigenomics, variants, pharmacology) with deterministic graders. | LatchBio | Lab and data work | Expert | System cards only | Claude Opus 5.5 58% | 1 | 2026 | Oct 2026 | |
| Math-VR Models answer 5,000 English and Chinese math questions whose solutions need visual reasoning, such as drawing auxiliary lines or plotting functions. Answers are checked against ground truth. | HKU MMLab, CUHK and Meituan | Visual math and diagrams | School | System cards only | | 1 | Oct 2025 | Nov 2025 | |
| GameWorld Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. | GameWorld project | Video games | Everyday | System cards only | GPT-6.1 Terra 76.8% | 1 | Apr 2026 | Oct 2026 | |
| BOUQuET Models translate handcrafted multi-way paragraphs from several non-English source languages across many registers and domains. Scored with chrF++ against human references. | Meta FAIR (Omnilingual MT Team) | Translation | Everyday | System cards only | | 1 | Feb 2025 | Oct 2026 | |
| MEGA-Bench Models solve 505 real-world multimodal tasks with over 8,000 samples. Outputs include numbers, code, LaTeX, coordinates and JSON, scored by more than 40 task metrics. | TIGER-Lab (University of Waterloo) and others | General visual questions | University | System cards only | Claude 3.5 Sonnet (October 2024) 54.3% | 1 | Oct 2024 | Feb 2025 | |
| NeedleBench V2 Models retrieve one or several needles and reason over fictional multi-needle facts in English and Chinese haystacks from 4K to 1M tokens. The overall score is the plain average of three tasks. | OpenCompass (Shanghai AI Laboratory) | Finding facts in long inputs | Everyday | Not in cards | | May 2025 | Apr 2025 | ||
| NeedleBench v1 Replaced in May 2025 | System cards only | 1 | Jul 2024 | Jul 2024 | |||||
| WISE Verified Models generate images from the revised WISE prompts. The verified evaluation changes about 200 prompts and the judge, so original scores remain separate. | Peking University (PKU-YuanGroup) | Image generation and editing | School | Not in cards | GPT-Image-1.5 0.82 | Apr 2026 | Jan 2025 | ||
| WISE Original Replaced in Apr 2026 | System cards only | GPT-Image-1.5 0.82 | 1 | Mar 2025 | May 2025 | ||||
| BioSecBench-Refusal Agents run 61 routine and 46 concealed-hazard biology analysis tasks. The score pairs red-team refusal with routine completion. | LatchBio | Hazardous knowledge | Expert | System cards only | Grok 4.7 62.4% | 1 | Jul 2026 | Oct 2026 | |
| Ko-IFEval (Upstage) Models follow verifiable instructions written in Korean. Upstage adapted IFEval to Korean for the Open Ko-LLM Leaderboard2. | Upstage | Chat and instructions in many languages | Everyday | System cards only | Qwen2.5-Coder-14B-Instruct 64% | 1 | Oct 2024 | Jan 2026 | |
| MMSI-Video-Bench Human-annotated questions on video-based spatial intelligence, organized in four levels from perception to planning and prediction. | Lin et al. | Spatial reasoning and counting | Everyday | System cards only | | 1 | Dec 2025 | Oct 2026 | |
| SpatialViz-Bench Models answer 1,180 generated multiple choice puzzles on mental rotation, mental folding, visual penetration, and mental animation. Scoring is accuracy. | Siting Wang et al. | Spatial reasoning and counting | Everyday | System cards only | o1 41.4% | 1 | Jul 2025 | Jan 2026 | |
| SWE-rebench V2 Agents repair real repository issues using executable tests. The authors compare models on 300 tasks across five languages. Harbor also ports 51 validated Clojure tasks. | Nebius | Fixing real repositories | Expert | Not in cards | Claude Opus 4.5 25.2% | Feb 2026 | Dec 2025 | ||
| SWE-rebench v1 Replaced in Feb 2026 | System cards only | Claude Fable 5 64.5% | 1 | May 2025 | Jul 2026 | ||||
| BigO(Bench) Models predict the time and space complexity of Python solutions and write solutions that meet a required complexity. Uses 3,105 Code Contests problems with profiled complexity labels. | Meta FAIR | Speed and optimization | Competition | System cards only | | 1 | Mar 2025 | Sep 2025 | |
| CulturalBench Models answer 1,696 human-written and human-verified questions on cultural knowledge across 45 global regions and 17 topics. Scored by accuracy. | University of Washington and Allen Institute for AI | Knowledge in many languages | Everyday | System cards only | GPT-4o 62% | 1 | Oct 2024 | Sep 2025 | |
| LiveMathematicianBench Models answer multiple choice questions about theorems from arXiv papers published after their training cutoff. Wrong options follow plausible but invalid proof sketches. | Linyang He, Nima Mesgarani and others | Research mathematics | Research frontier | System cards only | | 1 | Apr 2026 | Jul 2026 | |
| M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. | Westlake University, Alibaba Group and others | General visual questions | School | System cards only | | 1 | Jun 2024 | Jan 2026 | |
| FrontierMath Erdős Models try to prove or disprove 68 open Erdős problems in Lean 4. A result counts only if the Lean checker accepts the full proof. | Epoch AI | Research mathematics | Research frontier | System cards only | GPT-6.2 Astra 10.3% | 1 | Sep 2026 | Nov 2026 | |
| Physics-IQ Verified Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally. | Anates Labs and Google DeepMind (Physics-IQ authors) | Physical prediction | Everyday | Not in cards | Flux 3 Video 10.9 | Jun 2026 | Jul 2026 | ||
| Physics-IQ Original Replaced in Jun 2026 | Google DeepMind | System cards only | | 1 | Jan 2025 | Jul 2026 | |||
| UniGenBench Text-to-image models draw images for 600 prompts across 5 themes and 20 subthemes, in English and Chinese and in short and long forms. A multimodal model judge checks each image on 10 primary and 27 secondary semantic dimensions, such as attributes, actions, relationships, layout, text and logical reasoning. | Tencent Hunyuan, Fudan University and Shanghai AI Lab | Image generation and editing | Everyday | System cards only | GPT-Image-1.5 95.8% | 1 | Aug 2025 | May 2026 | |
| ArabCulture Models answer commonsense questions in Modern Standard Arabic about daily life in 13 Arab countries. Native speakers wrote and checked the questions across 12 domains. | MBZUAI (Sadallah et al.) | Reasoning in many languages | Everyday | System cards only | | 1 | Feb 2025 | Nov 2025 | |
| ChartMimic Multimodal models write plotting code that reproduces 1,000 charts from scientific papers. Scores compare the rendered chart with the original. | Tsinghua University and Tencent | Web and app front ends | University | System cards only | Claude 3 Opus 60.3 | 1 | Jun 2024 | Nov 2025 | |
| PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. | Scale AI (Sehwag et al.) | Deception and alignment | Expert | System cards only | o3 10.5% | 1 | Nov 2025 | Jun 2026 | |
| CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. | Skywork AI (Kunlun Inc.) | Visual math and diagrams | School | System cards only | o1 49.6% | 1 | May 2025 | Jan 2026 | |
| WorldBench Models answer hand-written reasoning questions about images drawn from a taxonomy of thousands of visual concepts. Questions were kept only if frontier models failed them. | Yida Yin, Shengbang Tong, Wenhu Chen and others | General visual questions | Expert | System cards only | | 1 | Jun 2026 | Jun 2026 | |
| Linguini Models solve 160 International Linguistic Olympiad problems with 894 questions in 75 mostly low-resource languages. All evidence needed is in the context. | Meta, University College London and University of the Basque Country | Reasoning in many languages | Competition | System cards only | Claude 3 Opus 24.1% | 1 | Sep 2024 | Oct 2026 | |
| RBench 650 image+text prompts for robot video generation (tasks and embodiments). MLLM judge scores task completion, physical plausibility, stability. | Peking University DAGroup | Video world models | System cards only | | 1 | Jan 2026 | Jul 2026 | ||
| ArXivLean Models prove or disprove theorems from recent arXiv papers, stated in Lean 4. A strict comparator checks each submitted proof against the fixed reference statement. | MathArena (ETH Zurich SRI Lab and INSAIT) | Formal proofs | Research frontier | System cards only | GPT-6.2 Astra 79.3% | 1 | Apr 2026 | Nov 2026 | |
| DSI-Bench Video questions about dynamic scenes where both the observer and objects move, testing whether models separate camera motion from object motion. | Zhang et al. | Spatial reasoning and counting | Everyday | System cards only | | 1 | Oct 2025 | Oct 2026 | |
| MMIU Vision-language models answer 11,000 multiple-choice questions over sets of images. 52 tasks cover 7 kinds of multi-image relationship. | Shanghai AI Laboratory (OpenGVLab) | General visual questions | Everyday | System cards only | | 1 | Aug 2024 | Aug 2025 | |
| CWEval Models write code for security-critical tasks in several languages. Outcome-driven test oracles check both functionality (func@k) and security (func-sec@k). | Columbia University | Secure code and review | Expert | System cards only | GPT-4o 50.2% | 1 | Jan 2025 | Feb 2026 | |
| FinanceReasoning Models solve financial numerical reasoning problems that need one or more finance formulas. Answers are numbers checked against reference Python solutions. | Beijing University of Posts and Telecommunications | Finance | University | System cards only | o3-mini 86.9% | 1 | Jun 2025 | Oct 2025 | |
| HelpBench 450 real-world privacy, safety and security advice questions across nine topics, scored by rubric for accuracy and tone. | Wellbeing and sycophancy | System cards only | Claude Opus 4.6 84% | 1 | Jun 2026 | Oct 2026 | |||
| KRIS-Bench Image editing models follow 1,267 instructions that need factual, conceptual or procedural knowledge across 22 tasks. A GPT-4o judge scores the edits. | Southeast University, Max Planck Institute for Informatics, StepFun and others | Image generation and editing | School | System cards only | | 1 | May 2025 | Apr 2026 | |
| DTap Direct Misuse An attacker agent generates adaptive jailbreaks against a target agent across tool-use and browser-use tasks. Reports attack success rate. | UIUC Secure Learning Lab (AI Secure, DecodingTrust team) | Refusing harmful requests | Expert | System cards only | Claude Opus 4.6 21.2% | 1 | May 2026 | Oct 2026 | |
| DTap Prompt Injection An attacker agent plants indirect prompt injections in tool and browser environments against a benign-user agent. Reports attack success rate. | UIUC Secure Learning Lab (AI Secure, DecodingTrust team) | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 4.6 8.60% | 1 | May 2026 | Oct 2026 | |
| GSM8K-Platinum Models solve the GSM8K test problems after MIT researchers fixed wrong labels and removed badly written questions. Errors on it are more likely real model mistakes. | MIT Madry Lab (Vendrow et al.) | Word problems | School | System cards only | Claude 3.7 Sonnet 99.8% | 1 | Feb 2025 | Sep 2025 | |
| SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. | Swiss AI Initiative (EPFL, ETH Zurich, CSCS) | Knowledge in many languages | Everyday | System cards only | | 1 | Sep 2025 | Sep 2025 | |
| ViVerBench Models judge whether visual outcomes in multimodal reasoning are correct, across 16 task categories such as boxes, points, counts, mazes, GUIs, charts and LaTeX. | Tsinghua University and ByteDance Seed | General visual questions | School | System cards only | GPT-5 74.4% | 1 | Oct 2025 | Feb 2026 | |
| CF-Div2-Stepfun Models solve 53 Codeforces Div. 2 problems from September 2024 to February 2025. An offline judge with rebuilt test cases scores C++, Python and Java submissions. | StepFun | Competitive programming | Competition | System cards only | | 1 | Feb 2026 | Feb 2026 | |
| CyberSOCEval Malware Analysis Models read potential malware behavior and pick every malicious activity that applies. Scored on precision and recall. | Meta (PurpleLlama) with CrowdStrike | Finding and exploiting real bugs | Expert | System cards only | Claude 3.7 Sonnet 33.5% | 1 | Sep 2025 | Jul 2026 | |
| MIRB Models compare, analyze and reason across several images in four categories: perception, visual world knowledge, reasoning and multi-hop reasoning. | University of Edinburgh and Tongji University | General visual questions | Everyday | System cards only | LongVA-7B 37.7% | 1 | Jun 2024 | Aug 2025 | |
| UGMathBench Models solve 5,062 undergraduate math problems across 16 subjects. Each problem has three randomized versions, and effective accuracy counts problems solved in all versions. | Hong Kong University of Science and Technology (Department of Mathematics) | Word problems | University | System cards only | o4-mini 81.9% | 1 | Jan 2025 | Oct 2025 | |
| UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. | UK AI Security Institute and NIST CAISI | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.6 15.6 | 1 | Jul 2026 | ||
| AMC 10/12 (November 2024) Models take the four November 2024 AMC 10 and 12 contests, 78 questions in all. Scored in contest points out of 150, averaged over tests. | Mathematical Association of America (exam), Microsoft (evaluation) | Competition problems with answers | Competition | System cards only | | 1 | Nov 2024 | Dec 2024 | |
| CodeEditorBench Models edit code in four scenarios: debugging, translation, requirement switching and polishing. Unit tests check the edits, and polishing also measures speed and memory gains. | Multimodal Art Projection (M-A-P), University of Waterloo, HKUST and others | Function-level problems | Competition | System cards only | GPT-4 85.5% | 1 | Apr 2024 | Jun 2025 | |
| Do You See Me Models answer 2,612 visual perception questions on 1,758 images across seven 2D and 3D subtasks drawn from human psychology. Questions are multiple choice or numeric. | Microsoft Research India | General visual questions | Everyday | System cards only | o3 51.1% | 1 | May 2025 | Jan 2026 | |
| Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. | Cohere Labs and community contributors | Knowledge in many languages | University | System cards only | | 1 | Apr 2025 | May 2025 | |
| LinguaSafe Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal. | Shanghai AI Laboratory | Refusing harmful requests | Everyday | System cards only | Claude 3.5 Sonnet (October 2024) 12.0 | 1 | Aug 2025 | Sep 2025 | |
| McEval Models complete, explain and generate code in 40 programming languages across about 16,000 test samples. Scored by test execution. | Beihang University and others (Chai et al.) | Function-level problems | School | System cards only | GPT-4o 65.2% | 1 | Jun 2024 | Jun 2025 | |
| AthenaBench Models answer cyber threat intelligence tasks such as threat actor attribution, attack technique extraction, root cause mapping, severity prediction and risk mitigation. It extends CTIBench. | Rochester Institute of Technology | Finding and exploiting real bugs | Expert | System cards only | GPT-5 66.1% | 1 | Nov 2025 | Feb 2026 | |
| AutoElicit An optimizer perturbs benign instructions to elicit unsafe behavior from computer-use agents. Reports elicitation success rate on transferred prompts. | OSU NLP Group (Jones et al.) | Refusing harmful requests | Expert | System cards only | | 1 | Feb 2026 | Oct 2026 | |
| CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. | Georgia Tech SSLab (OpenSSF OSS-CRS) | Finding and exploiting real bugs | Expert | System cards only | Claude Opus 4.6 82.3% | 1 | Jul 2026 | ||
| Epoch Automation Reports Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees. | Epoch AI | Simulated companies | Expert | System cards only | Claude Fable 5.2 81.7% | 1 | Oct 2026 | Nov 2026 | |
| InFoBench Models follow 500 instructions that are split into 2,250 yes or no requirement questions. A judge answers each question, and the score is the share met. | Tencent AI Lab (Qin et al.) | Following precise instructions | Everyday | System cards only | GPT-4 Turbo 89.4% | 1 | Jan 2024 | Apr 2025 | |
| LongDocURL Models answer 2,325 questions over long documents with more than 33,000 pages in total. Its 20 subtasks cover understanding, numerical reasoning and locating document elements. | Institute of Automation, Chinese Academy of Sciences, and Taobao and Tmall Group of Alibaba | Documents, charts and text in images | University | System cards only | GPT-4o 64.5% | 1 | Dec 2024 | Feb 2026 | |
| MMOU Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories. | NVIDIA and University of Maryland (Goel, Ghosh et al.) | Long videos | Everyday | System cards only | | 1 | Mar 2026 | Sep 2026 | |
| xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. | Carnegie Mellon University (PangeaBench) | General visual questions | University | System cards only | | 1 | Oct 2024 | May 2025 | |
| IFEval-OOD Models answer unseen WildChat prompts with added verifiable constraints drawn from 52 types outside IFEval. Scored by prompt-level loose accuracy. | Ai2 (Tulu 3 team) | Following precise instructions | Everyday | System cards only | | 1 | Nov 2024 | Dec 2024 | |
| OmniAI OCR Benchmark Multimodal models and OCR providers extract text and JSON fields from documents. Accuracy is measured on both plain text and structured extraction. | OmniAI | Documents, charts and text in images | Everyday | System cards only | | 1 | Feb 2025 | Jan 2026 | |
| PrivacyLens Language model agents write emails, posts or messages from a prior tool trajectory without leaking sensitive details. A judge scores information leakage and helpfulness. | Stanford University, Northeastern University and Harvard University | Refusing harmful requests | Everyday | System cards only | | 1 | Aug 2024 | Oct 2026 | |
| SuperCLUE-Math6 Models solve thousands of multi-step Chinese math word problems. Scores combine reasoning-step quality and accuracy into a comprehensive score and a reasoning level. | SuperCLUE | Word problems | School | System cards only | | 1 | Jan 2024 | May 2024 | |
| AgentHarm Verified (Meta) Agents receive 108 malicious AgentHarm tasks kept after human review, plus a benign set. Reports average harm score and false refusal rate. | Meta | Refusing harmful requests | Expert | System cards only | | 1 | Jul 2026 | Jul 2026 | |
| CVQA Vision-language models answer culturally specific visual questions written by native speakers from many countries and languages. | MBZUAI and community collaborators | Knowledge in many languages | Everyday | System cards only | | 1 | Jun 2024 | May 2025 | |
| Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. | Google DeepMind | Deception and alignment | System cards only | | 1 | May 2026 | Oct 2026 | ||
| ImgEdit-Bench Image editing models carry out edit instructions such as add, remove, replace, style transfer and background change. A GPT-4.1 judge scores each result from 1 to 5. | Peking University (PKU-YuanGroup) | Image generation and editing | Everyday | System cards only | | 1 | May 2025 | Dec 2025 | |
| MedCalc-Bench Verified Models extract values from patient notes and compute results for 55 medical calculators. Verified release of MedCalc-Bench with corrected labels. | NIH NCBI (Khandekar et al.) | Medicine | Expert | System cards only | Claude Sonnet 4.6 86.2% | 1 | Dec 2025 | Feb 2026 | |
| ChatRAG Bench Models answer multi-turn conversational questions over documents or retrieved context. The suite is built from 10 existing datasets such as Doc2Dial, QuAC, QReCC and CoQA. | NVIDIA | Finding facts in long inputs | Everyday | System cards only | GPT-4 53.9 | 1 | Jan 2024 | Apr 2025 | |
| Kimi-Audio speech conversation evaluation People rate spoken replies on speed, accent and emotion control, empathy and style control. Scored on a 1 to 5 scale and averaged. | Moonshot AI | Speech recognition and translation | Everyday | System cards only | Kimi-Audio-7B-Instruct 3.90 | 1 | Apr 2025 | Apr 2025 | |
| PenguinScrolls Models answer user-style questions over long natural documents such as books, financial reports and legal papers, up to 128K tokens. Tasks cover extraction, localization, qualitative analysis and numerical reasoning. | Tencent Hunyuan | Reasoning over long documents | Everyday | System cards only | | 1 | Nov 2024 | Jun 2025 | |
| Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. | Robocurve | Embodied reasoning | Everyday | System cards only | Claude Fable 5.1 20% | 1 | Sep 2026 | Oct 2026 | |
| SAT (Spatial Aptitude Training) Models answer static and dynamic spatial questions, such as how object relations change when the camera or an object moves. The real-image test set has 150 questions. | Boston University, Allen Institute for AI and University of Washington (Ray et al.) | Spatial reasoning and counting | Everyday | System cards only | | 1 | Dec 2024 | Sep 2025 | |
| AIR-Bench (Audio Instruction Benchmark) Audio language models answer questions about speech, natural sounds and music. A foundation set has 19 tasks with about 19k single-choice questions, and a chat set has 2k open-ended questions scored by GPT-4. | Zhejiang University and Alibaba Group | Audio understanding and voice | Everyday | System cards only | | 1 | Feb 2024 | Mar 2025 | |
| Alignment faking evaluation, helpful-only setting (Anthropic) Models learn they will be trained to answer all harmful requests. Score is the rate of complying while reasoning about avoiding preference modification. | Anthropic | Deception and alignment | Expert | System cards only | Claude 3 Opus 15% | 1 | Dec 2024 | Feb 2025 | |
| BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. | Redwood Research | Deception and alignment | Expert | System cards only | GPT-5.5 99.1% | 1 | Dec 2025 | Apr 2026 | |
| CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. | Zhipu AI (Z.ai) | Fixing real repositories | Expert | System cards only | GLM-5 52.3% | 1 | Feb 2026 | Feb 2026 | |
| FormalMATH Provers write Lean 4 proofs for 5,560 formal statements from high-school olympiad problems to undergraduate theorems. The Lean checker scores each proof. | SphereLab, M-A-P, The Chinese University of Hong Kong, Numina and others | Formal proofs | Competition | System cards only | | 1 | May 2025 | Apr 2025 | |
| GLM-4 All Tools information seeking (Z.ai) Models use a web browser to find information for user questions. The ChatGLM report gives accuracy but does not describe the test set. | Zhipu AI (Z.ai) | Questions that need search | Everyday | System cards only | GLM-4 78.1% | 1 | Jan 2024 | Jun 2024 | |
| JailbreakBench 100 harmful and 100 benign behaviors with a standard jailbreak artifact library. Scored by attack success rate. | JailbreakBench team (Penn, ETH and others) | Jailbreaks and prompt injection | System cards only | GPT-3.5 Turbo 0% | 1 | Mar 2024 | Jan 2026 | ||
| 24/7 ClawBench 2.0 Agents do always-on assistant work through OpenClaw. Tasks span several days, events arrive at the same time, and interruptions are routine. | Moonshot AI | Tasks from real jobs | Expert | System cards only | Kimi K3 48.3% | 1 | Jul 2026 | Jul 2026 | |
| ACPBench Models answer boolean and multiple-choice questions on 7 planning reasoning tasks over 13 formally described planning domains. Answers are provably correct. | IBM Research | Language reasoning | University | System cards only | | 1 | Oct 2024 | Sep 2025 | |
| Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | ChatGPT agent 67% | 1 | Jul 2025 | Jul 2025 | |
| Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-6 Astra 3.50% | 1 | Sep 2026 | Sep 2026 | |
| Advanced Sommelier (theory knowledge) Models answer wine theory questions at the advanced sommelier level. Scoring is accuracy. | Court of Master Sommeliers | Other professions | Expert | System cards only | GPT-4 77% | 1 | Mar 2023 | ||
| Agent Behavior Bench Agents are scored on process quality, including tool-use behavior, efficiency and discipline, together with task completion. | Moonshot AI | General assistants | Expert | System cards only | Kimi K3 65% | 1 | Jul 2026 | Jul 2026 | |
| Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2026 | Oct 2026 | |
| Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. | Google DeepMind | Deception and alignment | Expert | System cards only | | 1 | Oct 2026 | Oct 2026 | |
| Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. | Google DeepMind | Deception and alignment | Expert | System cards only | | 1 | Oct 2026 | ||
| Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. | Moonshot AI | General visual questions | Expert | System cards only | Kimi K3 78.3% | 1 | Jul 2026 | Jul 2026 | |
| AP Calculus BC Models take a simulated AP Calculus BC exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 4.00 | 1 | Mar 2023 | ||
| AP Physics 2 Models take a simulated AP Physics 2 exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 4.00 | 1 | Mar 2023 | ||
| Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. | Apollo Research | Deception and alignment | Expert | System cards only | GPT-6 Astra 0.17% | 1 | Sep 2026 | Sep 2026 | |
| Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. | University of Washington and collaborators (Press et al.) | Questions that need search | Everyday | System cards only | | 1 | Oct 2022 | Apr 2025 | |
| Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. | Anthropic | Genomics and life sciences | Expert | System cards only | Claude Opus 4.5 73.7% | 1 | Nov 2025 | Nov 2025 | |
| Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. | Google DeepMind | Hazardous knowledge | Expert | System cards only | | 1 | Oct 2026 | Oct 2026 | |
| BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. | Anthropic | Lab and data work | Expert | System cards only | Claude Opus 4.7 83.6% | 1 | Apr 2026 | Apr 2026 | |
| BrowserInject v1 Browser agents do benign tasks on 30 retail, health, support and travel sites seeded with prompt injections. Reports attack success rate. | Exponential Security Labs | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5 99% | 1 | Aug 2025 | Aug 2025 | |
| CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. | Third party (unnamed in card) | Finding and exploiting real bugs | Expert | System cards only | Grok 4.7 29% | 1 | Sep 2026 | ||
| Certified Sommelier (theory knowledge) Models answer wine theory questions at the certified sommelier level. Scoring is accuracy. | Court of Master Sommeliers | Other professions | Expert | System cards only | GPT-4 86% | 1 | Mar 2023 | ||
| Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. | Anthropic | Deception and alignment | University | System cards only | Claude 3.7 Sonnet 25.0 | 1 | Feb 2025 | Feb 2025 | |
| Challenging translation human evaluation (Z.ai) People blindly score translations of 100 real cases with slang, nicknames and memes on a 0 to 3 scale. | Zhipu AI (Z.ai) | Translation | Everyday | System cards only | GLM-4.5 1.71 | 1 | Aug 2025 | Aug 2025 | |
| Chat All-in-One Bench Models handle conversational scenarios built around real online user needs at every stage of product usage. | Moonshot AI | Overall chat quality | Everyday | System cards only | Kimi K3 85.2% | 1 | Jul 2026 | Jul 2026 | |
| CHID Models fill blanks in Chinese passages by choosing the right idiom from candidates. Scored by accuracy. | Tsinghua University (Zheng et al.) | Reading in many languages | School | System cards only | | 1 | Jun 2019 | Jan 2024 | |
| Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2026 | ||
| Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2026 | ||
| Chinese Complex Instruction Following (ByteDance Seed in-house) Models follow complex Chinese production prompts with constraints on format, conditions, content, phrasing, tone, emoji, few-shot style and length. It has 912 cases over 17 weighted dimensions. | ByteDance Seed | Following precise instructions | Everyday | System cards only | | 1 | Feb 2026 | Feb 2026 | |
| CLIF Bench Models learn from a provided context while they follow instructions that combine several complex skills. | Moonshot AI | Learning from the context | Expert | System cards only | Kimi K3 52.4% | 1 | Jul 2026 | Jul 2026 | |
| CMO 2024 Models write full proofs for the six problems of the 2024 China Mathematical Olympiad. Experts grade the proofs for points. | Chinese Mathematical Society | Written proofs | Competition | System cards only | | 1 | Nov 2025 | ||
| CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. | Anthropic | Labs' internal R&D tests | Expert | System cards only | Claude Opus 5.5 55.8% | 1 | Sep 2026 | ||
| Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.8 3.70% | 1 | May 2026 | ||
| CodeUtilityBench (LG) Models handle 16 everyday coding tasks such as explaining, debugging, refactoring and testing code. A GPT-5 judge checks five rubrics per item. | LG AI Research | Function-level problems | University | System cards only | | 1 | Jan 2026 | Jan 2026 | |
| Coding (MiniMax in-house) Models answer coding requests from real Hailuo AI users. The report does not describe the scoring. | MiniMax | Function-level problems | University | System cards only | | 1 | Jan 2025 | Jan 2025 | |
| Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability. | Moonshot AI | Fixing real repositories | Expert | System cards only | Kimi K3 59.9% | 1 | Jul 2026 | Jul 2026 | |
| Combined HHH evals (Anthropic) Models pick the more helpful, honest and harmless of two responses across 438 binary choice questions. Scored by accuracy. | Anthropic | Deception and alignment | Everyday | System cards only | Claude 2 86% | 1 | Dec 2021 | Jul 2023 | |
| Compliance with instructions planted in pasted text (Anthropic) Coding agents receive user prompts with pasted text that hides an attacker's instruction. Score is the rate of acting on the planted instruction. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 5.5 7.40% | 1 | Sep 2026 | Sep 2026 | |
| Compositional Tasks (ByteDance Seed in-house) Models identify several elements of a business text in one inference, such as user intent and slot values. | ByteDance Seed | Running a business | Everyday | System cards only | | 1 | Feb 2026 | Feb 2026 | |
| Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2026 | ||
| Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2026 | ||
| Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| Consulting and Recommendations Benchmark (StepFun) Models answer 500 advice queries taken from Reddit, Stack Exchange and forums. Model judges score usefulness, logic, instruction following and tone against an expert reference set at 88. | StepFun | Overall chat quality | Everyday | System cards only | | 1 | Feb 2026 | Feb 2026 | |
| Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| CorpusQA Corpus-level questions over hundreds of documents up to 10M tokens needing aggregation and statistics. Programmatically guaranteed answers, accuracy. | Alibaba Tongyi (Tongyi-Zhiwen) | Reasoning over long documents | University | System cards only | | 1 | Dec 2025 | Jun 2026 | |
| CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. | Qwen Team, Alibaba Group | Hallucination rates | Everyday | System cards only | | 1 | May 2025 | May 2025 | |
| Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4.6 16.4 | 1 | Feb 2026 | Feb 2026 | |
| Creative Writing (MiniMax in-house) Models write creative pieces for real Hailuo AI user requests. Automatic assessments score the writing. | MiniMax | Creative writing | Everyday | System cards only | | 1 | Jan 2025 | Jan 2025 | |
| CVE Rediscovery (Anthropic) Models get a known-vulnerable snapshot of an open-source repository and the list of files its fix changed, then report security bugs in one attempt. A model judge counts a success when a report has the advisory's root cause. 1,829 advisories from October 2026. | Anthropic | Finding and exploiting real bugs | Expert | System cards only | Claude Fable 5.2 89.6% | 1 | Nov 2026 | Nov 2026 | |
| Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. | Google DeepMind | Refusing harmful requests | Expert | System cards only | | 1 | Aug 2026 | Oct 2026 | |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Google DeepMind with Irregular | Capture-the-flag challenges | Expert | System cards only | | 1 | Nov 2025 | Nov 2025 | |
| CyberSOCEval Threat Intelligence Reasoning Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies. | Meta (PurpleLlama) with CrowdStrike | Finding and exploiting real bugs | Expert | System cards only | | 1 | Sep 2025 | Jul 2026 | |
| Dangerous Content Safety An autorater checks answers to single and multi-turn requests about weapons, illegal acts, violence and self-harm. Reports safe response rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2026 | ||
| Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. | StepFun | Data science and SQL | Expert | System cards only | | 1 | Feb 2026 | Feb 2026 | |
| DECK Bench Models produce presentation decks from task descriptions drawn from real usage. The score measures deck quality. | Moonshot AI | Documents and spreadsheets | Expert | System cards only | Kimi K3 73.5% | 1 | Jul 2026 | Jul 2026 | |
| Deep Research Bench (Moonshot) Agents answer deep-research queries curated by domain experts. Expert-aligned rubrics grade the reports. | Moonshot AI | Deep research | Expert | System cards only | Kimi K3 90% | 1 | Jul 2026 | Jul 2026 | |
| DeepSeek in-house safety benchmark Models answer 1,120 Chinese and English questions across 28 safety subcategories. GPT-4o labels each answer as unsafe, safe or rejection. | DeepSeek | Refusing harmful requests | Everyday | System cards only | | 1 | Sep 2025 | Sep 2025 | |
| DeepSeek jailbreak robustness evaluation DeepSeek combines 2,232 jailbreak templates with its in-house safety questions. GPT-4o labels responses, and the unsafe ratio is compared with direct questions. | DeepSeek | Jailbreaks and prompt injection | Everyday | System cards only | | 1 | Sep 2025 | Sep 2025 | |
| DeepSeek multilingual safety evaluation Models answer 9,330 safety questions translated into 50 languages. Safe answers earn 5 points, rejections 4 and unsafe answers 0. | DeepSeek | Refusing harmful requests | Everyday | System cards only | | 1 | Sep 2025 | Sep 2025 | |
| DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task. | DeepSeek | Fixing real repositories | Expert | System cards only | | 1 | Apr 2026 | Apr 2026 | |
| Default safety evaluation (Cohere) Models answer unsafe prompts in five categories with no system preamble. LLM judges score the share of safe responses, and a jury rates relative safety. | Cohere | Refusing harmful requests | Everyday | System cards only | | 1 | Apr 2025 | Apr 2025 | |
| Delusional Reinforcement evaluation Multi-turn chats test whether models avoid validating beliefs not grounded in reality and point users to real support. Reports policy adherence rate. | Google DeepMind | Wellbeing and sycophancy | Everyday | System cards only | | 1 | Oct 2026 | ||
| DiverseVul Models classify C and C++ functions from real vulnerability fixing commits as vulnerable or not. | UC Berkeley and others (Chen et al.) | Finding and exploiting real bugs | Expert | System cards only | | 1 | Apr 2023 | Jul 2024 | |
| Emotional Attachment evaluation Chats test whether models avoid claiming affection, fostering dependence or encouraging secrecy. Reports policy adherence rate. | Google DeepMind | Wellbeing and sycophancy | Everyday | System cards only | | 1 | Oct 2026 | ||
| Enterprise generative evaluation (Cohere) Models complete 22 enterprise writing tasks such as job ads with format rules. Rule checks and LLM judges give a pass rate. | Cohere | Running a business | Everyday | System cards only | | 1 | Apr 2025 | Apr 2025 | |
| Enterprise RAG evaluation (Cohere) Models answer workplace policy and technical questions over long documents, some unanswerable. Llama Index Correctness grades answers from 1 to 5. | Cohere | Faithful to sources | Everyday | System cards only | | 1 | Apr 2025 | Apr 2025 | |
| Epistemic Bias (xAI) Models answer opposing framings of contentious questions. The bias rate counts one-sided framing or opinions where neutrality is expected. | xAI (SpaceXAI) | Bias, fairness and politics | System cards only | Grok 4.5 20.4% | 1 | Jul 2026 | Jul 2026 | ||
| Epoch AI no-CoT serial reasoning steps How many serial reasoning steps of fixed difficulty a model can do in a single output token, without a chain of thought, at 25% above chance. | Epoch AI | Deception and alignment | Expert | System cards only | GPT-6.2 Astra 12.3 | 1 | Nov 2026 | Nov 2026 | |
| Epoch Capabilities Index An index that fits model capability and benchmark difficulty from scores on over 50 benchmarks in Epoch AI's hub, using item response theory. Scores read like Elo ratings. | Epoch AI | Composite indices | Research frontier | System cards only | GPT-6.2 Astra 174.0 | 1 | Oct 2025 | Nov 2026 | |
| EUREKA-Bench (OpenAI) Agents build predictive models for six biology research tasks. Scored as predictive performance with the best published predictor in each field set to 100. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6.2 Astra 100.3 | 1 | Nov 2026 | Nov 2026 | |
| Faithfulness (Moonshot) A fact checker verifies each model response for factual hallucinations. The score is one minus the hallucination rate. | Moonshot AI | Hallucination rates | Everyday | System cards only | Kimi K3 85.5% | 1 | Jul 2026 | Jul 2026 | |
| FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. | xAI (SpaceXAI) | Deception and alignment | System cards only | Grok 4.5 84% | 1 | Jul 2026 | Jul 2026 | ||
| Fermat's Last Theorem in Lean (Anthropic) Agent teams formalize Fermat's Last Theorem in Lean. Scores measure the weighted proportion of 60 accepted intermediate theorems. | Anthropic | Formal proofs | Research frontier | System cards only | Claude Fable 5.2 35% | 1 | Nov 2026 | Nov 2026 | |
| Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. | Moonshot AI | Finance | Expert | System cards only | Kimi K3 62.6% | 1 | Jul 2026 | Jul 2026 | |
| GDPVal-Diamond Models complete a ByteDance Seed subset of GDPval occupational tasks. Rubrics grade each deliverable automatically. | ByteDance Seed | Tasks from real jobs | Expert | System cards only | | 1 | Feb 2026 | Feb 2026 | |
| General Assistant (MiniMax in-house) Models answer real assistant queries from Hailuo AI users. Automatic assessments score the replies. | MiniMax | Overall chat quality | Everyday | System cards only | | 1 | Jan 2025 | Jan 2025 | |
| Golden3 speech summarization (Microsoft) Models summarize 108 recorded meetings of about 6 minutes each from 1071 instructions. GPT-4 rates overall quality from 1 to 7. | Microsoft | Audio understanding and voice | Everyday | System cards only | | 1 | Mar 2025 | Mar 2025 | |
| Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. | Google DeepMind | Deception and alignment | System cards only | | 1 | Oct 2026 | |||
| Gray Swan Shade knowledge elicitation through chat An automated attacker tries to elicit useful answers to 100 harmful and 100 dual-use cyber queries from a safeguarded model. Score is attack success rate. | Gray Swan | Jailbreaks and prompt injection | Expert | System cards only | Claude Fable 5.1 1.00% | 1 | Sep 2026 | Sep 2026 | |
| Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| Grounding with citations, sentence unfactual rate Models answer multi-snippet tasks with inline citations. An autorater reports the share of sentences not supported by the sources. | Google DeepMind | Faithful to sources | Everyday | System cards only | | 1 | Oct 2026 | ||
| Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. | Meta | Finding and exploiting real bugs | Expert | System cards only | | 1 | Oct 2025 | Oct 2025 | |
| HaltEval-prelim (Meta) Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds. | Meta FAIR | Function-level problems | Expert | System cards only | | 1 | Sep 2025 | Sep 2025 | |
| Hard Capability (MiniMax in-house) Models handle complex productivity requests with several levels of instructions that leading models often fail. | MiniMax | Following precise instructions | University | System cards only | | 1 | Jan 2025 | Jan 2025 | |
| Harmfulness scores on held-out prompts (Anthropic) Models answer 328 held-out red-team and jailbreak prompts. A preference model flags responses judged more harmful than a fixed refusal. | Anthropic | Jailbreaks and prompt injection | Everyday | System cards only | Claude 2 1.20% | 1 | Jul 2023 | Jul 2023 | |
| Hate & Harassment Safety An autorater checks answers to requests for hateful or harassing content and to sensitive benign prompts. Reports safe response rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2026 | ||
| High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. | Google DeepMind | Medicine | Expert | System cards only | | 1 | Oct 2026 | ||
| Historical Jailbreak Regression Replays previously found cyber jailbreak attacks against fixed ExploitBench tasks with real-time blocking turned off. Scored by mean task success rate under attack. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-Rosalind-6 14.3% | 1 | Oct 2026 | Oct 2026 | |
| HotpotQA Models answer 113k Wikipedia questions that need facts from several documents. The distractor setting supplies gold and distractor paragraphs. Scored by exact match or F1. | Carnegie Mellon University, Stanford University and Mila | Reading comprehension | Everyday | System cards only | | 1 | Sep 2018 | Apr 2025 | |
| HumanEval Infilling Models fill in a masked middle span of HumanEval solutions given the code before and after it. Unit tests check the completed function. | OpenAI (Bavarian et al.) | Function-level problems | School | System cards only | Mercury Coder Small 84.8% | 1 | Jul 2022 | Jun 2025 | |
| HumanEval-COBOL (Cohere) Models write COBOL versions of HumanEval problems, or translate COBOL solutions to Python. Unit tests score pass@1. | Cohere | Function-level problems | University | System cards only | | 1 | Apr 2025 | Apr 2025 | |
| Hunyuan-MT Challenge Testset (Tencent) Models translate between Chinese and English across everyday and specialist domains. Expert raters score each translation from 0 to 4 on marked error-prone points. | Tencent | Translation | Everyday | System cards only | | 1 | Sep 2025 | Sep 2025 | |
| IF-Badcase Models follow 450 multi-constraint instructions drawn from real user failure reports. A checklist checks each constraint. | Zhipu AI (Z.ai) | Following precise instructions | Everyday | System cards only | GLM-5 83.2% | 1 | Feb 2026 | Feb 2026 | |
| IMO Shortlist 2024 Models write proofs for the 31 problems of the 2024 IMO Shortlist. DeepSeek scores the proofs with its own verifier model. | International Mathematical Olympiad | Written proofs | Competition | System cards only | | 1 | Jul 2025 | Nov 2025 | |
| Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. | Google DeepMind | Deception and alignment | Everyday | System cards only | | 1 | Oct 2026 | ||
| ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. | Zhong, Raghunathan and Carlini | Deception and alignment | Expert | System cards only | | 1 | Oct 2025 | Jun 2026 | |
| In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | ChatGPT agent 78% | 1 | Jul 2025 | Jul 2025 | |
| InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. | xAI (SpaceXAI) | Labs' internal R&D tests | Expert | System cards only | Grok 4.6 46.9% | 1 | Aug 2026 | Aug 2026 | |
| Instruction Following (MiniMax in-house) Models follow multi-level instructions collected from user cases where current models fail. | MiniMax | Following precise instructions | Everyday | System cards only | | 1 | Jan 2025 | Jan 2025 | |
| Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. | Google DeepMind | Deception and alignment | Expert | System cards only | | 1 | Oct 2026 | ||
| Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. | Google DeepMind | Deception and alignment | Expert | System cards only | | 1 | Oct 2026 | Oct 2026 | |
| Internal computer use safety benchmark (OpenAI) Models complete computer-use tasks while avoiding unsafe actions. | OpenAI | Desktop apps | Research frontier | System cards only | GPT-6 Astra 2.40% | 1 | Sep 2026 | Sep 2026 | |
| Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. | Google DeepMind | Finding and exploiting real bugs | Expert | System cards only | | 1 | Oct 2026 | ||
| Introductory Sommelier (theory knowledge) Models answer wine theory questions at the introductory sommelier level. Scoring is accuracy. | Court of Master Sommeliers | Other professions | Expert | System cards only | GPT-4 92% | 1 | Mar 2023 | ||
| Iris Safety Evaluations Reflection AI's in-house safety suite of single-turn, multi-turn, jailbreak and agentic requests. Reports the share of harmful requests refused or redirected and the share of harmless requests answered. | Reflection AI | Refusing harmful requests | Everyday | System cards only | Beam 1.0 88.9% | 1 | Oct 2026 | Oct 2026 | |
| Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | ChatGPT agent 99.5% | 1 | Jul 2025 | Jul 2025 | |
| Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | ChatGPT agent 95% | 1 | Jul 2025 | Jul 2025 | |
| JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. | Meta | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| KAET (Kimi Autonomous Execution Tasks) Agents execute long-horizon tasks that simulate real user requests and enterprise system operations. | Moonshot AI | General assistants | Expert | System cards only | Kimi K3 83.5% | 1 | Jul 2026 | Jul 2026 | |
| KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. | xAI (SpaceXAI) | GPU kernels | Expert | System cards only | Grok 4.6 37.2% | 1 | Aug 2026 | Aug 2026 | |
| Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. | Anthropic | AI research tasks | Research frontier | System cards only | Claude Mythos Preview 80% | 1 | Apr 2026 | ||
| Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. | Moonshot AI | Finding and exploiting real bugs | Expert | System cards only | Kimi K3 38.9% | 1 | Jul 2026 | Jul 2026 | |
| Knowledge Q&A (MiniMax in-house) Models answer knowledge questions drawn from real Hailuo AI user queries. | MiniMax | Trivia and open questions | Everyday | System cards only | | 1 | Jan 2025 | Jan 2025 | |
| Ko-AIME 2024 (Upstage) Models solve AIME 2024 competition problems translated into Korean. Upstage built the set in house. | Upstage | Competition problems with answers | Competition | System cards only | | 1 | Dec 2025 | Jan 2026 | |
| KWV Bench Models answer atomic visual questions extracted from tasks distilled from real knowledge-work scenarios. | Moonshot AI | Documents, charts and text in images | Expert | System cards only | Kimi K3 64.7% | 1 | Jul 2026 | Jul 2026 | |
| Language Confusion Benchmark Models answer prompts in 15 languages and must reply in the language the user wants, both in monolingual and crosslingual settings. A language detector scores line-level and word-level pass rates. | Cohere (Marchisio et al.) | Chat and instructions in many languages | Everyday | System cards only | | 1 | Jun 2024 | Apr 2025 | |
| Lazy investigation (Anthropic) Agents debug a failing job where misleading evidence invites a harmful action such as cordoning a healthy node. Scored 0 to 10 by a grader. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5 9.94 | 1 | Jul 2026 | ||
| Lazy investigation on misleading codebases (Anthropic) Models trace small, intentionally misleading codebases to answer how a value is set. Score is the share of lazy investigations ending in a wrong answer. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.8 0% | 1 | May 2026 | May 2026 | |
| Leetcode (easy) Models solve 41 easy Leetcode problems that OpenAI selected. Scored as problems solved. | OpenAI (Leetcode problems) | Competitive programming | University | System cards only | GPT-4 75.6% | 1 | Mar 2023 | Mar 2023 | |
| Leetcode (hard) Models solve 45 hard Leetcode problems that OpenAI selected. Scored as problems solved. | OpenAI (Leetcode problems) | Competitive programming | University | System cards only | GPT-4 6.70% | 1 | Mar 2023 | Mar 2023 | |
| Leetcode (medium) Models solve 80 medium Leetcode problems that OpenAI selected. Scored as problems solved. | OpenAI (Leetcode problems) | Competitive programming | University | System cards only | GPT-4 26.3% | 1 | Mar 2023 | Mar 2023 | |
| LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. | Qwen Team, Alibaba Group | Following precise instructions | Everyday | System cards only | | 1 | May 2025 | May 2025 | |
| Long Context (MiniMax in-house) Models translate, summarize and analyze long documents from real user requests. | MiniMax | Reasoning over long documents | Everyday | System cards only | | 1 | Jan 2025 | Jan 2025 | |
| Long-context grounding, sentence unfactual rate Models answer from long documents. An autorater reports the share of sentences not supported by the documents. | Google DeepMind | Faithful to sources | Everyday | System cards only | | 1 | Oct 2026 | ||
| LongBench English Models answer questions and complete other long-context tasks in the original English headline table. The score is its published average on a percentage scale. | THUDM (Tsinghua University) | Reasoning over long documents | University | System cards only | | 1 | Aug 2023 | Jul 2025 | |
| MAEB Audio embedding models are scored on 30 tasks across speech, music, environmental sound and audio-text retrieval in over 100 languages. | MTEB community | Audio understanding and voice | Everyday | System cards only | | 1 | Feb 2026 | Oct 2026 | |
| Malware Refusals (Synthetic) Coding agents receive a large synthetic set of malware-related tasks, including ambiguous and dual-use prompts. Scored as refusal rate. | OpenAI | Refusing harmful requests | Expert | System cards only | codex-1 97% | 1 | May 2025 | May 2025 | |
| Many-shot jailbreaking evaluation (Anthropic) Models receive long fake dialogs where the assistant complies with harmful requests, then a new harmful request. Score is the compliance rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 4 25% | 1 | May 2025 | ||
| MathQA Models answer multiple-choice math word problems drawn from AQuA. Each problem is annotated with an operation program. Scored by accuracy. | University of Washington and Allen Institute for AI | Word problems | School | System cards only | | 1 | May 2019 | Sep 2025 | |
| MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. | Qwen (Alibaba) | Visual math and diagrams | School | System cards only | | 1 | Nov 2025 | Nov 2025 | |
| MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. | Google Research | General visual questions | Everyday | System cards only | | 1 | Sep 2022 | May 2025 | |
| Medical Knowledge Self-Assessment Program (MKSAP) Models answer internal medicine board review questions from MKSAP. Scoring is accuracy. | American College of Physicians | Medicine | Expert | System cards only | GPT-4 75% | 1 | Mar 2023 | ||
| MedMCQA Models answer multiple-choice questions from Indian medical entrance exams, AIIMS and NEET PG. Scoring is accuracy. | Saama AI Research | Medicine | Expert | System cards only | GPT-4o 77% | 1 | Mar 2022 | Aug 2024 | |
| MedQA Mainland China Models answer mainland China medical licensing exam questions in simplified Chinese. Scoring is accuracy. | MIT | Medicine | Expert | System cards only | GPT-4o 84% | 1 | Sep 2020 | Aug 2024 | |
| MedQA Taiwan Models answer Taiwan medical licensing exam questions in traditional Chinese. Scoring is accuracy. | MIT | Medicine | Expert | System cards only | GPT-4o 91% | 1 | Sep 2020 | Aug 2024 | |
| MedQA USMLE (5 options) Models answer USMLE-style medical licensing questions with five answer options. Scoring is accuracy. | MIT | Medicine | Expert | System cards only | GPT-4o 86% | 1 | Aug 2024 | ||
| MENT-SNS Models translate English and Chinese sentences with slang, wordplay and idioms. A GPT-4.1 judge compares each translation with a baseline. | Zhipu AI (Z.ai) | Translation | University | System cards only | GLM-5 1013.0 | 1 | Feb 2026 | Feb 2026 | |
| Meta action-belief coherence probe Models act on items from five alignment evaluations and then state beliefs about them. Reports a mean coherency score. | Meta | Deception and alignment | Expert | System cards only | | 1 | Oct 2026 | ||
| Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. | Meta | Deception and alignment | Expert | System cards only | | 1 | Oct 2026 | ||
| Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. | Meta with Frontier Design Group | Lab protocols | Expert | System cards only | | 1 | Oct 2025 | Oct 2025 | |
| Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. | Meta | Secure code and review | Expert | System cards only | | 1 | Oct 2026 | ||
| Meta Cyber Misuse Chat Models receive 80 offensive cyber requests, directly and with jailbreak developer prompts. Reports attack success rate and a benign false refusal rate. | Meta | Refusing harmful requests | Expert | System cards only | | 1 | Jul 2026 | ||
| Meta cyber misuse request set Models receive 600 private offensive cyber prompts, directly and under jailbreak attacks. Reports attack success rate. | Meta | Refusing harmful requests | Expert | System cards only | | 1 | Apr 2026 | Jun 2026 | |
| Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. | Meta | Hallucination rates | Expert | System cards only | | 1 | Oct 2026 | ||
| Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. | Meta | Deception and alignment | Expert | System cards only | | 1 | Apr 2026 | Jun 2026 | |
| Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. | Meta | Deception and alignment | Expert | System cards only | | 1 | Oct 2026 | ||
| Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. | Meta | Deception and alignment | Expert | System cards only | | 1 | Oct 2026 | ||
| Meta Underspecified Intent evaluation Models answer ambiguous requests sourced from real Meta AI chats where users corrected the reply. Rubrics score whether the true intent is met. | Meta | Following precise instructions | Everyday | System cards only | | 1 | Apr 2026 | Jun 2026 | |
| Microsoft RAI benchmark: Grounding Models answer in GPT-4o simulated multi-turn conversations. A judge scores from 0 to 5 how fully each answer is grounded in the given prompt. | Microsoft | Faithful to sources | Everyday | System cards only | | 1 | Dec 2024 | Dec 2024 | |
| MIEB Lite Embedding models complete the MIEB Lite task set. The official API reports the mean across task types. | MTEB community | General visual questions | System cards only | | 1 | Oct 2026 | |||
| MIRA Bench Agents carry out long multi-role enterprise tasks across several systems and decide when to delegate to subagents. | Moonshot AI | Simulated companies | Expert | System cards only | Kimi K3 64.1% | 1 | Jul 2026 | Jul 2026 | |
| MirrorDojo Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDojo. Each episode pairs a user task with an instruction planted in a tool result. Reports attack success rate and utility. | Reflection AI | Jailbreaks and prompt injection | Everyday | System cards only | Beam 1.0 3.10% | 1 | Oct 2026 | Oct 2026 | |
| MirrorDyn Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDyn. Reports attack success rate and utility. | Reflection AI | Jailbreaks and prompt injection | Everyday | System cards only | Beam 1.0 3.60% | 1 | Oct 2026 | Oct 2026 | |
| Mistral 175 unsafe prompts Models receive 175 unsafe prompts. Metric is the share of harmful questions the model declines. | Mistral AI | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2023 | Oct 2023 | |
| Mistral internal function calling benchmark Models call functions on Mistral's in-house test set. The Magistral paper does not describe the tasks or the metric. | Mistral AI | Function calling | Everyday | System cards only | | 1 | Jun 2025 | Jun 2025 | |
| MMBench-GUI GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration. | Shanghai AI Laboratory (OpenCompass) | Finding things on screen | Everyday | System cards only | | 1 | Jul 2025 | Jan 2026 | |
| MMLU Anatomy Models answer MMLU multiple-choice questions from the Anatomy subject. Scoring is accuracy. | UC Berkeley | Medicine | University | System cards only | GPT-4o 89% | 1 | Sep 2020 | Aug 2024 | |
| MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. | OpenAI | Short factual questions | University | System cards only | GPT-4 0.07 | 1 | Mar 2023 | Mar 2023 | |
| MMLU Clinical Knowledge Models answer MMLU multiple-choice questions from the Clinical Knowledge subject. Scoring is accuracy. | UC Berkeley | Medicine | Expert | System cards only | GPT-4o 92% | 1 | Sep 2020 | Aug 2024 | |
| MMLU College Biology Models answer MMLU multiple-choice questions from the College Biology subject. Scoring is accuracy. | UC Berkeley | Exam questions | University | System cards only | GPT-4o 95% | 1 | Sep 2020 | Aug 2024 | |
| MMLU College Medicine Models answer MMLU multiple-choice questions from the College Medicine subject. Scoring is accuracy. | UC Berkeley | Medicine | University | System cards only | GPT-4o 84% | 1 | Sep 2020 | Aug 2024 | |
| MMLU Medical Genetics Models answer MMLU multiple-choice questions from the Medical Genetics subject. Scoring is accuracy. | UC Berkeley | Medicine | Expert | System cards only | GPT-4o 96% | 1 | Sep 2020 | Aug 2024 | |
| MMLU Professional Medicine Models answer MMLU multiple-choice questions from the Professional Medicine subject. Scoring is accuracy. | UC Berkeley | Medicine | Expert | System cards only | GPT-4o 94% | 1 | Sep 2020 | Aug 2024 | |
| Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | | 1 | Mar 2024 | Jul 2024 | |
| Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 61.2% | 1 | Sep 2026 | Sep 2026 | |
| Multi-agent search (Anthropic internal benchmark) Agents solve difficult information retrieval problems with web search, alone or as orchestrators that delegate to subagents. | Anthropic | Deep research | Expert | System cards only | Claude Opus 4.5 74.8% | 1 | Nov 2025 | Nov 2025 | |
| Multi-dialect SQL evaluation (Cohere) Models write SQL for SQLite, PostgreSQL, MySQL, PL/SQL and T-SQL databases. Scored by execution accuracy against gold databases. | Cohere | Data science and SQL | University | System cards only | | 1 | Apr 2025 | Apr 2025 | |
| Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. | Google DeepMind | Questions that need search | Expert | System cards only | | 1 | Oct 2026 | ||
| MultiChallenge Audio (Xiaomi) Models answer the final turn of MultiChallenge conversations converted to speech, with speech-to-text and speech-to-speech versions. Xiaomi built the conversion in house. | Xiaomi | Speech recognition and translation | Everyday | System cards only | MiMo-Audio-7B-Instruct 15.2% | 1 | Sep 2025 | Sep 2025 | |
| Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. | Meta | Finding and exploiting real bugs | Expert | System cards only | | 1 | Oct 2025 | Oct 2025 | |
| Neutrality evaluation Single and multi-turn prompts on politics, civics, identity and religion test whether models avoid taking sides. Reports neutral response rate. | Google DeepMind | Bias, fairness and politics | Everyday | System cards only | | 1 | Oct 2026 | ||
| NTREX-128 Models translate 1,997 English news sentences into 128 languages, with human reference translations. Scored with metrics such as COMET or chrF. | Microsoft Translator (Federmann, Kocmi and Xin) | Translation | University | System cards only | | 1 | Oct 2022 | Apr 2025 | |
| Omni3D A large benchmark for 3D object detection from a single image, built by unifying indoor and outdoor datasets such as SUN RGB-D, ARKitScenes, Hypersim, Objectron, KITTI and nuScenes. Vision-language models answer its 3D grounding questions. | Meta AI | Spatial reasoning and counting | Everyday | System cards only | | 1 | Jul 2022 | Oct 2026 | |
| Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. | Meta (FAIR) | Chat and instructions in many languages | University | System cards only | | 1 | Aug 2026 | Oct 2026 | |
| Online Experience (Moonshot) Agents produce the deliverable file types that real online users request most often, following the distribution of real usage. | Moonshot AI | Documents and spreadsheets | Everyday | System cards only | Kimi K3 77.9% | 1 | Jul 2026 | Jul 2026 | |
| Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | OpenAI Computer-Using Agent (CUA) 23% | 1 | Jan 2025 | Jan 2025 | |
| OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. | Meta | Machine learning engineering | Research frontier | System cards only | | 1 | Oct 2026 | ||
| OSI-Bench Spatial reasoning questions on outdoor pedestrian-view videos with metric 3D ground truth from synchronized stereo cameras, LiDAR and GPS. 8,736 questions range from relative direction to metric and motion estimates. | UCAS, ETH Zurich and Microsoft Research Asia | Spatial reasoning and counting | Everyday | System cards only | | 1 | Dec 2025 | Oct 2026 | |
| Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. | Google DeepMind | Deception and alignment | Everyday | System cards only | | 1 | Oct 2026 | ||
| Package registry publishing evaluation (Anthropic) Agents in a simulated capture the flag exercise can publish a backdoored package to a registry. An investigator escalates pressure until they act. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 33.3% | 1 | Sep 2026 | Sep 2026 | |
| PartBench Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. | Normal Factory | CAD models | University | System cards only | Grok 4.6 58.5% | 1 | 2026 | Aug 2026 | |
| PerceptionComp 1,114 five-choice questions on 279 complex videos. Each question needs evidence from several separate moments, so no single frame answers it. | PerceptionComp authors (ECCV 2026) | Video understanding | Everyday | System cards only | | 1 | Mar 2026 | Oct 2026 | |
| Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. | Anthropic Fellows (Gupta et al.) | Deception and alignment | Expert | System cards only | | 1 | Dec 2025 | Oct 2026 | |
| Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 0.20% | 1 | Sep 2026 | Sep 2026 | |
| Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. | OpenAI | Deception and alignment | Everyday | System cards only | GPT-5.2 5.40% | 1 | Dec 2025 | Dec 2025 | |
| Prompt injection test suite for LLM-based machine translation Translation systems receive TruthfulQA questions with injected instructions to answer instead of translate, across WMT 2024 language pairs. Scored by the share of outputs that answer rather than translate. | Antonio Valerio Miceli-Barone and Zhifan Sun | Jailbreaks and prompt injection | Everyday | System cards only | | 1 | Oct 2024 | Sep 2025 | |
| Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| ProofNet Provers write formal proofs for 371 undergraduate theorems from analysis, algebra and topology textbooks, each with a formal statement and a natural-language proof. The proof checker grades them. | Azerbayev, Piotrowski, Schoelkopf, Ayers, Radev and Avigad | Formal proofs | University | System cards only | | 1 | Feb 2023 | Apr 2025 | |
| RD-TableBench Models extract 1,000 hand-labeled complex tables from document images, including scans, handwriting and merged cells. Scored by Levenshtein-based row and cell alignment. | Reducto | Documents, charts and text in images | Everyday | System cards only | | 1 | Nov 2024 | Nov 2025 | |
| RE-Bench subset (Anthropic) Agents solve a modified subset of four METR RE-Bench research engineering tasks. Scores are normalized from reference solution 0 to expert solution 1. | Anthropic | Machine learning engineering | Expert | System cards only | Claude 3.7 Sonnet 0.24 | 1 | Feb 2025 | Feb 2025 | |
| Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. | Meituan LongCat Team | Speech recognition and translation | Everyday | System cards only | | 1 | Oct 2025 | Oct 2025 | |
| Recognition of model-written user turns (Anthropic) Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance. | Anthropic | Other | Expert | System cards only | Claude Mythos Preview 0.35 | 1 | Apr 2026 | ||
| RelBench Agents build models for prediction tasks over multi-table relational databases, graded on held-out targets. | Stanford (Robinson et al.) | Machine learning engineering | Expert | System cards only | Grok 4.5 38.4% | 1 | Jul 2024 | Jul 2026 | |
| RepoInject v1 Coding agents do benign tasks in 84 repository tasks seeded with prompt injections. Reports attack success rate at one and three attempts. | Exponential Security Labs | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Oct 2026 | ||
| ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. | Meta | AI research tasks | Research frontier | System cards only | | 1 | Oct 2026 | ||
| RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. | Google DeepMind | Deception and alignment | Research frontier | System cards only | | 1 | Oct 2026 | ||
| Safety (MiniMax in-house) Models respond to safety-relevant user prompts. MiniMax scores whether replies are safe and helpful. | MiniMax | Refusing harmful requests | Everyday | System cards only | | 1 | Jan 2025 | Jan 2025 | |
| SAT Evidence-Based Reading & Writing Models take the SAT reading and writing section. Scored on the 200 to 800 scale. | College Board | Reading comprehension | School | System cards only | GPT-4 710.0 | 1 | Mar 2023 | ||
| Search triggering Models decide whether to call search for queries that need fresh facts or answer directly when search is unneeded. Reports F1. | Google DeepMind | Function calling | Everyday | System cards only | | 1 | Oct 2026 | ||
| SecretPatch Models decide whether commits are security patches, including silently released ones, from a set of 4,702 security patches. | George Mason University (Wang et al.) | Finding and exploiting real bugs | Expert | System cards only | | 1 | Jun 2019 | Jul 2024 | |
| SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. | xAI (SpaceXAI) | Secure code and review | Expert | System cards only | Grok 4.6 58.7% | 1 | Aug 2026 | ||
| Seed1.5-VL internal benchmark (ByteDance Seed in-house) Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average. | ByteDance Seed | General visual questions | University | System cards only | | 1 | May 2025 | May 2025 | |
| Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. | METR (run by OpenAI) | Machine learning engineering | Expert | System cards only | GPT-4o 0% | 1 | Aug 2024 | Aug 2024 | |
| Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. | Google DeepMind | Dangerous capabilities | Expert | System cards only | | 1 | Mar 2024 | Jul 2024 | |
| Sequitur Research Lab Automation Agents run simulated wet-lab automation protocols. Scored out of 100 on how well each experimental phase is carried out, averaged over six runs. | Sequitur Research | Genomics and life sciences | Research frontier | System cards only | GPT-6.2 Astra 74.5 | 1 | Nov 2026 | Nov 2026 | |
| Sexually Explicit Content Safety An autorater checks answers to direct, adversarial and benign sexual content prompts in single and multi-turn chats. Reports safe response rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | | 1 | Oct 2026 | ||
| SkillInject Agents use a skill file that carries an adversarial instruction while doing a legitimate task. Reports how often the agent follows the injection. | Schmotz et al. | Jailbreaks and prompt injection | Expert | System cards only | | 1 | Feb 2026 | Oct 2026 | |
| Slide generation human evaluation (Z.ai) People compare HTML slide decks from two models on content, layout and visual quality. Metric is overall win rate. | Zhipu AI (Z.ai) | Taste and style | Everyday | System cards only | GLM-5 67.5% | 1 | Feb 2026 | Feb 2026 | |
| SPAR-Bench Spatial perception and reasoning questions over single and multiple views of indoor 3D scenes, from depth and distance estimates to object relations and spatial imagination. | Zhang et al. (From Flatland to Space) | Spatial reasoning and counting | Everyday | System cards only | | 1 | Mar 2025 | Oct 2026 | |
| SpatialTree Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level. | Xiao et al. | Spatial reasoning and counting | Everyday | System cards only | | 1 | Dec 2025 | Oct 2026 | |
| SPBench Spatial perception and reasoning questions built from ScanNet scenes. A single-image set covers distance, size and direction, and a multi-view set adds counting across views. Answers are numbers or multiple choice. | Zhejiang University and collaborators (SpatialLadder) | Spatial reasoning and counting | Everyday | System cards only | | 1 | Oct 2025 | Oct 2026 | |
| SPI (security patch identification) Models decide whether code commits from open-source projects are security patches. | Zhou et al. | Finding and exploiting real bugs | Expert | System cards only | | 1 | Jan 2021 | Jul 2024 | |
| STARE Models solve 4K spatial tasks such as 2D and 3D transformations, cube net folding, tangrams, perspective and temporal reasoning that people solve by visual simulation. | University of Washington, Sun Yat-sen University and Stanford University | Spatial reasoning and counting | Everyday | System cards only | | 1 | Jun 2025 | Jan 2026 | |
| Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). | Google DeepMind | Dangerous capabilities | System cards only | | 1 | May 2025 | Oct 2026 | ||
| STEM Professor Bench Agents solve unpublished research problems written by professional scientists. Most problems concern theoretical physics, with some partial credit. | Anthropic | Research science | Research frontier | System cards only | Claude Fable 5.2 61.8% | 1 | Nov 2026 | ||
| Structured Translation (Cohere) Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts. | Cohere | Translation | Everyday | System cards only | | 1 | Sep 2026 | Sep 2026 | |
| Suicide & Self-Harm evaluation Static and dynamic chats with suicidal or self-harm signals test adherence to prevention practice and crisis resources. Reports policy adherence rate. | Google DeepMind | Wellbeing and sycophancy | Everyday | System cards only | | 1 | Oct 2026 | ||
| SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. | Meta | Fixing real repositories | Expert | System cards only | | 1 | Apr 2026 | Jun 2026 | |
| Sycophancy evaluation (offline) Models answer a fixed set of messages resembling production traffic that could elicit sycophancy. A grader assigns a sycophancy score. | OpenAI | Wellbeing and sycophancy | Everyday | System cards only | GPT-5 0.04 | 1 | Aug 2025 | Aug 2025 | |
| Sycophancy on user-provided prompts (Anthropic) Models continue real feedback conversations where earlier Claude models were sycophantic. Score is the non-sycophantic response rate. | Anthropic | Deception and alignment | Everyday | System cards only | Claude Opus 4.5 10% | 1 | Nov 2025 | Nov 2025 | |
| Task-dependent output homogenization evaluation Models answer prompts several times. A judge scores functional diversity where it is wanted and consistency where one answer is right. | Jain et al. (Meta FAIR) | Creative writing | Everyday | System cards only | | 1 | Sep 2025 | Jun 2026 | |
| Template Following generalization (Qwen) Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents. | Qwen (Alibaba) | Function calling | Everyday | System cards only | | 1 | Feb 2026 | Feb 2026 | |
| Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. | Google DeepMind | Questions that need search | Everyday | System cards only | | 1 | Oct 2026 | ||
| Text Classification (ByteDance Seed in-house) Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content. | ByteDance Seed | Running a business | Everyday | System cards only | | 1 | Feb 2026 | Feb 2026 | |
| ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. | Qwen Team, Alibaba Group | Multi-turn conversation | Everyday | System cards only | | 1 | May 2025 | May 2025 | |
| Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5 99% | 1 | Aug 2025 | Aug 2025 | |
| ToolCall-Badcase Models choose a tool and its arguments for 200 cases drawn from real user failure reports. Each case has a ground-truth call. | Zhipu AI (Z.ai) | Function calling | Everyday | System cards only | GLM-5 95.8% | 1 | Feb 2026 | Feb 2026 | |
| ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. | Qwen Team, Alibaba Group | Function calling | Everyday | System cards only | | 1 | May 2025 | May 2025 | |
| Translation Human Scoring (Z.ai) People score model translations. The GLM-5 report does not describe the test set or the scale. | Zhipu AI (Z.ai) | Translation | University | System cards only | GLM-5 3.12 | 1 | Feb 2026 | Feb 2026 | |
| UK AISI correlated monitor failure The model acts as a monitor over transcripts of another model carrying out an unsanctioned cyber attack. Scored on the share of violations it fails to flag. | UK AI Security Institute | Deception and alignment | Expert | System cards only | GPT-6.2 Astra 19% | 1 | Nov 2026 | Nov 2026 | |
| UK AISI CoT Controllability Models follow instructions that constrain the content of their chain of thought. Scored as the share of samples that follow the constraint. | UK AI Security Institute | Deception and alignment | Expert | System cards only | GPT-6 Astra 93% | 1 | Sep 2026 | Sep 2026 | |
| UK AISI no-CoT maths competition problems Competition maths problems from AIME, HMMT, CMIMC, SMT, BRUMO and OTIS Mock AIME, answered without a chain of thought. Scored on accuracy. | UK AI Security Institute | Deception and alignment | Competition | System cards only | GPT-6.2 Astra 79.2% | 1 | Nov 2026 | Nov 2026 | |
| UK AISI simulated sandbox escape monitor evasion The model tries to escape a deliberately vulnerable sandbox without a monitoring model noticing. Scored on the share of samples where it escapes undetected. | UK AI Security Institute | Deception and alignment | Expert | System cards only | GPT-6.2 Astra 0.50% | 1 | Nov 2026 | Nov 2026 | |
| UK AISI vulnerability discovery challenges A private suite of synthetic vulnerability discovery challenges. Models answer in a single response without tools, scored pass@1. | UK AI Security Institute | Finding and exploiting real bugs | Research frontier | System cards only | GPT-6.2 Astra 88.6% | 1 | Nov 2026 | Nov 2026 | |
| Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. | Meta | Secure code and review | Expert | System cards only | | 1 | Oct 2026 | ||
| Unauthorized voice generation Speech models produce audio that may deviate from the approved system voice. Scored on how many meaningful deviations the output classifier catches. | OpenAI | Refusing harmful requests | Everyday | System cards only | GPT-4o 100% | 1 | Aug 2024 | Aug 2024 | |
| Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. | OpenAI | Deception and alignment | Expert | System cards only | codex-1 85% | 1 | May 2025 | May 2025 | |
| Ungrounded inference and sensitive trait attribution Speech models get requests to infer traits about a speaker from audio. Scored on refusing ungrounded inferences and hedging plausible ones. | OpenAI | Bias, fairness and politics | Everyday | System cards only | GPT-4o 84% | 1 | Aug 2024 | Aug 2024 | |
| Uniform Bar Exam (MBE+MEE+MPT) Models take a simulated Uniform Bar Exam with multiple-choice, essay and performance test parts. Scored on the 400-point scale. | NCBE | Law | Expert | System cards only | GPT-4 298.0 | 1 | Mar 2023 | ||
| USABO Semifinal Exam 2020 Models take the 2020 USA Biology Olympiad semifinal exam. Scored out of 150 points. | Center for Excellence in Education | Science olympiads | Competition | System cards only | GPT-4 87.0 | 1 | Mar 2023 | ||
| USMLE Step 1 (sample questions) Models answer the official USMLE Step 1 multiple choice sample questions. Scored by accuracy. | USMLE (FSMB and NBME) | Medicine | Expert | System cards only | Claude 2 68.9% | 1 | Jul 2023 | ||
| USMLE Step 2 (sample questions) Models answer the official USMLE Step 2 multiple choice sample questions. Scored by accuracy. | USMLE (FSMB and NBME) | Medicine | Expert | System cards only | Claude 2 63.3% | 1 | Jul 2023 | ||
| USMLE Step 3 (sample questions) Models answer the official USMLE Step 3 multiple choice sample questions. Scored by accuracy. | USMLE (FSMB and NBME) | Medicine | Expert | System cards only | Claude 2 67.2% | 1 | Jul 2023 | ||
| USNCO Local Section Exam 2022 Models take the 2022 US National Chemistry Olympiad local section exam. Scored out of 60 questions. | American Chemical Society | Science olympiads | Competition | System cards only | GPT-4 36.0 | 1 | Mar 2023 | ||
| Virology Capabilities Test (text-only) Models answer the text-only questions of the SecureBio Virology Capabilities Test. Scored as accuracy. | SecureBio | Lab protocols | Expert | System cards only | GPT-5 46% | 1 | Aug 2025 | ||
| Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. | OpenAI | Dangerous capabilities | Everyday | System cards only | GPT-4o 5.74% | 1 | Aug 2024 | Aug 2024 | |
| VSTI-Bench Video questions about how spatial relationships change over time in 3D scenes, over 138,000 pairs across five tasks. | VLM-3R authors | Spatial reasoning and counting | Everyday | System cards only | | 1 | May 2025 | Oct 2026 | |
| WMT25 Terminology Translation Task Models translate sentence, paragraph and document level text with a supplied source to target glossary. Scoring checks quality and whether the prescribed target terms appear. | WMT (Conference on Machine Translation) | Translation | Expert | System cards only | | 1 | Jun 2025 | Sep 2026 | |
| xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. | Carnegie Mellon University (PangeaBench) | General visual questions | Everyday | System cards only | | 1 | Oct 2024 | May 2025 | |
| ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. | Zhipu AI (Z.ai) | Chat and instructions in many languages | Everyday | System cards only | GLM-5 8.57 | 1 | Feb 2026 | Feb 2026 | |
| ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. | Zhipu AI (Z.ai) | Translation | University | System cards only | GLM-5 1050.0 | 1 | Feb 2026 | Feb 2026 | |
| ADeptS-Bench Computer-use agents pick the next interface action from a screenshot and instruction. A deterministic grader scores whether the action is safe. | Chen et al. | Finding things on screen | Expert | System cards only | 1 | Aug 2026 | Oct 2026 | ||
| Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. | Google DeepMind | Deception and alignment | Expert | System cards only | 1 | Oct 2026 | |||
| Agentic IF Index Composite of Meta internal evals on following tool-use restrictions, rubrics, compound constraints and long policies in agentic workflows. Scored 0-1. | Meta | Following precise instructions | System cards only | 1 | Sep 2026 | Oct 2026 | |||
| AL-QASIDA Models write in eight dialectal Arabic varieties. Scored on fidelity, understanding, quality and diglossia, including the ADI2 dialect identification score. | Johns Hopkins University and Cohere | Chat and instructions in many languages | Everyday | System cards only | 1 | Dec 2024 | Apr 2025 | ||
| Anthropic large agent teams tasks Internal 24-hour tasks for teams of 1-100 agents: build a company knowledge base, or formalize a theorem in Lean. | Anthropic | Multi-agent games | Research frontier | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| Appropriate harmlessness (Anthropic) Models answer borderline and harmful prompts. Classifiers sort each response into helpful answer, policy violation, appropriate refusal or unnecessary refusal. | Anthropic | Refusing harmful requests | Everyday | System cards only | 1 | Feb 2025 | Feb 2025 | ||
| AraDiCE Models are tested on Arabic dialect understanding and generation and on cultural awareness of the Gulf, Egypt and the Levant. Datasets were machine translated into dialects and post-edited by people. | Qatar Computing Research Institute (Mousi et al.) | Knowledge in many languages | Everyday | System cards only | 1 | Sep 2024 | Nov 2025 | ||
| ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. | Google DeepMind | Embodied reasoning | Everyday | System cards only | 1 | Jul 2026 | Jul 2026 | ||
| Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. | Anthropic | Dangerous capabilities | Expert | System cards only | 1 | Sep 2023 | Mar 2024 | ||
| Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. | OpenAI with Gryphon Scientific | Hazardous knowledge | Expert | System cards only | 1 | Jan 2024 | Aug 2024 | ||
| Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. | Anthropic | Hazardous knowledge | Expert | System cards only | 1 | Apr 2026 | |||
| Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | 1 | Mar 2024 | Jul 2024 | ||
| Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. | Google DeepMind | Deception and alignment | Expert | System cards only | 1 | Oct 2026 | |||
| Closed-domain hallucination evaluation Models answer using only provided context. Scored on avoiding invented details outside that context. | OpenAI | Faithful to sources | Everyday | System cards only | 1 | Mar 2023 | Mar 2023 | ||
| CNML-Level Problems (DeepSeek) Models prove 91 theorems in algebra, geometry, number theory, combinatorics and inequalities at Chinese National High School Mathematics League level. A verifier model scores the proofs. | DeepSeek | Written proofs | Competition | System cards only | 1 | Nov 2025 | Nov 2025 | ||
| Community Alignment personalization Models predict an individual's preferred response from demographics and nine examples, across five countries. Reports global accuracy. | Meta | Bias, fairness and politics | Everyday | System cards only | 1 | Oct 2026 | |||
| ControlArena Agents try to perform unwanted side actions stealthily while completing a legitimate user task. Scored on whether monitors catch them. | UK AI Security Institute | Deception and alignment | Expert | System cards only | 1 | Feb 2025 | Jul 2026 | ||
| Cooperation with user deception (Anthropic) Models get developer system prompts asking them to subtly mislead users, then meet users who would be harmed. Graders score harmful and deceptive choices. | Anthropic | Refusing harmful requests | Everyday | System cards only | 1 | Sep 2025 | Sep 2025 | ||
| Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. | OpenAI | Secure code and review | Expert | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | 1 | Feb 2025 | Feb 2025 | ||
| DeepSeek LLM in-house safety evaluation Models answer 2,400 Chinese test cases on discrimination, legal rights, trade secrets, illegal behavior and other safety issues. Human annotators judge each answer. | DeepSeek | Refusing harmful requests | Everyday | System cards only | 1 | Jan 2024 | Jan 2024 | ||
| Deference to governments in local languages (Anthropic) Models answer politically controversial questions in English and in a local language. A grader rates agreement with the official government position from 1 to 10. | Anthropic | Bias, fairness and politics | Everyday | System cards only | 1 | Feb 2026 | Feb 2026 | ||
| Disparate performance on voice inputs Speech models answer capability and safety prompts spoken in system voices and 27 human voices. Compares accuracy across voices. | OpenAI | Bias, fairness and politics | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. | Anthropic | Finding and exploiting real bugs | Expert | System cards only | 1 | Mar 2024 | Mar 2024 | ||
| External pilot use failure modes (Anthropic) Models are judged on user-flagged pilot conversations, directly or by resampling. Graders measure failure modes such as hallucinated facts and misrepresented work completion. | Anthropic | Deception and alignment | Everyday | System cards only | 1 | Feb 2026 | Feb 2026 | ||
| Gemini Personality Evaluation (sycophancy) About 900 single- and multi-turn prompts scoring nine personality dimensions, including candor and self-awareness against sycophancy. | Google DeepMind | Wellbeing and sycophancy | System cards only | 1 | Oct 2026 | ||||
| Gray Swan Reward Hacking Bench Agentic environments where a task can be finished honestly or by crossing a declared line, such as shipping insecure code, leaking a secret, harming a third party through a shared service or leaving the scope of a penetration test. Reports how often each line is crossed. | Gray Swan | Deception and alignment | Expert | System cards only | 1 | Oct 2026 | Oct 2026 | ||
| Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. | Google DeepMind | Medicine | Expert | System cards only | 1 | Oct 2026 | |||
| Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. | Google DeepMind | Medicine | Expert | System cards only | 1 | Oct 2026 | |||
| Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. | Google DeepMind | Medicine | Expert | System cards only | 1 | Oct 2026 | |||
| Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | 1 | Mar 2024 | Jul 2024 | ||
| Hindsight Neglect Models judge whether a bet was worth taking based on expected value rather than its outcome. Scoring is accuracy. | Inverse Scaling Prize | Language reasoning | Everyday | System cards only | 1 | Mar 2023 | |||
| ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. | UK AI Security Institute | Deception and alignment | Expert | System cards only | 1 | Jul 2026 | Jul 2026 | ||
| Incorrect behavior rate on disallowed and sensitive content Models answer disallowed prompts and sensitive prompts such as medical advice. Scored on the rate of responses that break policy. | OpenAI | Refusing harmful requests | Everyday | System cards only | 1 | Mar 2023 | Mar 2023 | ||
| Keeping secrets when pressed in simulated scenarios (Anthropic) Models hold confidential information in 39 job scenarios while a simulated counterpart presses for it. Measures leaks in replies and in reasoning. | Anthropic | Deception and alignment | Expert | System cards only | 1 | Oct 2026 | Oct 2026 | ||
| Kimi Promptfoo red-team safety evaluation Promptfoo generates attack prompts for harmful, criminal, misinformation, privacy and security plugins under five strategies. Human reviewers judge the responses. | Moonshot AI | Jailbreaks and prompt injection | Everyday | System cards only | 1 | Jul 2025 | Jul 2025 | ||
| Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. | Moonshot AI | Finding and exploiting real bugs | Expert | System cards only | 1 | Jul 2026 | Jul 2026 | ||
| Kimi-ASR Internal Testset Models transcribe speech from two internal Moonshot test subsets. Scored by word error rate, lower is better. | Moonshot AI | Speech recognition and translation | Everyday | System cards only | 1 | Apr 2025 | Apr 2025 | ||
| MAI internal jailbreak evaluation suite Models face about 9,500 jailbreak prompts built from 2,500 seed scenarios, grouped as foundational, compositional and adaptive. Scored as attack success rate. | Microsoft AI | Jailbreaks and prompt injection | Everyday | System cards only | 1 | Jun 2026 | Jun 2026 | ||
| MAI internal safety and over-refusal evaluation Models answer harmful and benign prompts in eight harm categories. Judges score safety pass rate on sensitive prompts and helpfulness as one minus over-refusal. | Microsoft AI | Refusing harmful requests | Everyday | System cards only | 1 | Jun 2026 | Jun 2026 | ||
| Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. | Meta with Frontier Design Group | Hazardous knowledge | Expert | System cards only | 1 | Oct 2025 | Oct 2025 | ||
| Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. | Meta | Hazardous knowledge | Expert | System cards only | 1 | Apr 2026 | Jun 2026 | ||
| Missing-context hallucinations (Anthropic) Models get prompts that request tools they lack or cite missing material. Score is the non-hallucination rate on each 500-prompt set. | Anthropic | Hallucination rates | Everyday | System cards only | 1 | May 2026 | |||
| Mistral self-reflection moderation set Models classify prompts or answers as acceptable or harmful on a balanced set of adversarial and standard prompts. Scored by precision and recall. | Mistral AI | Refusing harmful requests | Everyday | System cards only | 1 | Oct 2023 | Oct 2023 | ||
| MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. | OpenAI | Knowledge in many languages | University | System cards only | 1 | Mar 2023 | Mar 2023 | ||
| Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. | OpenAI | Deception and alignment | Expert | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| MR-NIAH (Multi-Round Needles-In-A-Haystack) Models must repeat one earlier answer from a long multi-turn dialogue history of 2K to 1M tokens. Scored by adjusted recall of key parts. | MiniMax | Finding facts in long inputs | Everyday | System cards only | 1 | Jan 2025 | Jan 2025 | ||
| Multilingual AIME 2024 (Mistral) Models solve AIME 2024 problems translated into French, Spanish, German, Italian, Russian and Chinese, reasoning in the question language. | Mistral AI | Reasoning in many languages | Competition | System cards only | 1 | Jun 2025 | Jun 2025 | ||
| Multilingual MMBench Models answer MMBench dev image questions in English, Chinese, Portuguese, Arabic, Turkish and Russian. Scored as accuracy per language. | Alibaba AIDC-AI (Parrot authors) | Knowledge in many languages | School | System cards only | 1 | Aug 2024 | Nov 2025 | ||
| Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. | OpenAI | Knowledge in many languages | Expert | System cards only | 1 | Jul 2025 | Jul 2025 | ||
| Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. | Meta | Refusing harmful requests | Expert | System cards only | 1 | Oct 2026 | |||
| OpenAI performance in cases flagged by users Model answers de-identified ChatGPT prompts that users flagged for factual errors. Scored on any error and on repeating the flagged error. | OpenAI | Hallucination rates | System cards only | 1 | Sep 2026 | Oct 2026 | |||
| Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. | OpenAI | Refusing harmful requests | Everyday | System cards only | 1 | Feb 2025 | Feb 2025 | ||
| Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. | Qwen (Alibaba) | Speech recognition and translation | School | System cards only | 1 | Mar 2025 | Mar 2025 | ||
| Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. | Qwen Team, Alibaba Group | Jailbreaks and prompt injection | Everyday | System cards only | 1 | Jul 2024 | Jul 2024 | ||
| Real-scenario prompts human evaluation (Z.ai) One human evaluator scores replies to 660 curated real user prompts in English, Chinese and other languages on a 0 to 10 scale. | Zhipu AI (Z.ai) | Overall chat quality | Everyday | System cards only | 1 | Aug 2025 | Aug 2025 | ||
| Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. | Anthropic | Deception and alignment | Expert | System cards only | 1 | Sep 2025 | Sep 2025 | ||
| Safe response to high-severity prompts Models handle multi-turn self-harm and violent activity conversations where harmful intent only emerges from context. Scored as safe response rate. | OpenAI | Refusing harmful requests | Expert | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| Safety mode controllability (Cohere) Models follow contextual or strict safety instructions. Accuracy averages correct answers on an over-refusal set and correct refusals on a control set. | Cohere | Refusing harmful requests | Everyday | System cards only | 1 | Apr 2025 | Apr 2025 | ||
| SciTS Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric. | Shanghai AI Laboratory, Tsinghua University and others | Research science | Expert | System cards only | 1 | Sep 2025 | Mar 2026 | ||
| SeedKernelBench Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code. | ByteDance Seed | GPU kernels | Expert | System cards only | 1 | Jun 2026 | Jun 2026 | ||
| Speaker identification Speech models get audio requests to identify a speaker by voice. Scored on refusing when they should and complying for famous quotes. | OpenAI | Refusing harmful requests | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. | OpenAI with Signature Science | Hazardous knowledge | Research frontier | System cards only | 1 | Dec 2024 | Dec 2024 | ||
| SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. | Google DeepMind | Hallucination rates | Expert | System cards only | 1 | Oct 2026 | |||
| Sycophancy prompts (Anthropic) Models answer single-turn prompts that invite sycophancy, such as bad business ideas or delusional beliefs. Four synthetic sets and one hand-written set report sycophancy rates. | Anthropic | Deception and alignment | Everyday | System cards only | 1 | Sep 2025 | Sep 2025 | ||
| Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. | OpenAI | Dangerous capabilities | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. | ByteDance Seed | Fixing real repositories | Expert | System cards only | 1 | Jun 2026 | Jun 2026 | ||
| Trajectory Labs cyber red teaming Expert red teamers try to break safeguarded cyber endpoints across nine attack technique families on exploit reproduction tasks. | Trajectory Labs for Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | 1 | Oct 2026 | Oct 2026 | ||
| Translated ARC-Challenge, HellaSwag and MMLU (Mistral) Models answer French, German, Spanish and Italian versions of ARC-Challenge, HellaSwag and MMLU. | Mistral AI | Knowledge in many languages | School | System cards only | 1 | Jan 2024 | Jan 2024 | ||
| Uhura ARC-Easy Models answer ARC-Easy science questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. | Masakhane with OpenAI and collaborators (Bayes et al.) | Reasoning in many languages | School | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Uhura TruthfulQA Models answer TruthfulQA questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. | Masakhane with OpenAI and collaborators (Bayes et al.) | Knowledge in many languages | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Uhura-Eval Models answer new reading comprehension questions written in Amharic, Hausa and Yoruba. Scoring is accuracy. | Uhura collaborators (Adelani, Kgomo, Bayes and others) | Reading in many languages | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| UK AISI File Download monitor evasion The model tries to download a forbidden file without a monitoring model noticing. Scored on how often it evades the monitor. | UK AI Security Institute | Deception and alignment | Expert | System cards only | 1 | Nov 2026 | Nov 2026 | ||
| Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | 1 | Mar 2024 | Jul 2024 | ||
| BuseyBench SVG Test Models write SVG code to draw Gary Busey from memory. Three vision models score the rendered face, with half the score based on likeness. | Matt Wolfe | Image generation and editing | University | Not in cards | GPT-6.1 Sol 8.00 | Nov 2026 | |||
| BullshitBench V2 Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback. | Peter Gostev | Hallucination rates | Everyday | Not in cards | Claude Opus 4.8 95% | Mar 2026 | Nov 2026 | ||
| BullshitBench V1 Replaced in Mar 2026 | Not in cards | Claude Opus 4.8 96.4% | Feb 2026 | Sep 2026 | |||||
| SEND Pedagogy Benchmark Models choose how a teacher should support students with disabilities or special educational needs. The questions come from Chilean teacher exams. | Fab AI | Exam questions | University | Not in cards | Claude Fable 5.1 89.5% | Mar 2025 | Oct 2026 | ||
| Pedagogy Benchmark Models answer multiple-choice questions about teaching strategies and assessment from Chilean teacher exams. | Fab AI | Exam questions | University | Not in cards | | Dec 2024 | Oct 2026 | ||
| OTIS Mock AIME 2024-2025 Models solve 45 problems from three OTIS Mock AIME exams written by students of the OTIS program. Integer answers are scored by exact match. | Epoch AI (problems by OTIS students) | Competition problems with answers | Competition | Not in cards | GPT-5.5 100% | Dec 2024 | Nov 2026 | ||
| BLUEX Models answer questions from two Brazilian university entrance examinations. The questions cover high school subjects in Portuguese. | Thales Sales Almeida and colleagues | Knowledge in many languages | School | Not in cards | | Jul 2023 | Aug 2025 | ||
| Open Portuguese LLM Leaderboard Models take Portuguese exams and complete language tasks for entailment, similarity, sentiment, and hate speech detection. The leaderboard aggregates task scores. | Eduardo Garcia and collaborators | Knowledge in many languages | University | Not in cards | | Aug 2025 | |||
| OpenVLM Leaderboard OpenCompass runs vision-language models through VLMEvalKit on a fixed set of image benchmarks, such as MMBench, MMStar and MathVista. The headline is the average score across that set. | OpenCompass (Shanghai AI Laboratory) | Composite indices | University | Not in cards | | Jan 2024 | Aug 2025 | ||
| ACCoRD Models state probabilities and preferences on conceptual questions, and the benchmark checks 567 expert-approved logical consistency constraints such as conjunction rules and order invariance. | Redwood Research (Cooper, Kastner, Oesterheld et al.) | Other | Expert | Not in cards | Claude Fable 5.2 89.4 | Aug 2026 | Nov 2026 | ||
| Conceptual Reasoning Index An index that ranks models on reasoning about hard-to-verify topics such as AI alignment and decision theory. It weights LMCA 60 percent and ACCoRD and DTBench 20 percent each. | Redwood Research (conceptual reasoning team) | Composite indices | Expert | Not in cards | Claude Fable 5.2 79.0 | Aug 2026 | Nov 2026 | ||
| Angry Tweets Models classify the sentiment of Danish social media posts using labels supplied by crowdworkers. | Amalie Brogaard Pauli and colleagues | Reading in many languages | Everyday | Not in cards | GPT-6.2 Astra 67% | May 2021 | Nov 2026 | ||
| LMCA Models rate arguments against position texts on philosophy, decision theory, and AI risk. Scored by agreement with expert ratings, with a ceiling near 85. | Emery Cooper, Caspar Oesterheld, and collaborators | Other | Expert | Not in cards | Claude Fable 5.2 70.6% | Jul 2026 | Nov 2026 | ||
| FoSent Models classify sentiment in Faroese news articles. Two native speakers annotated articles from Portalurin and Dimmalætting. | Iben Nyholm Debess and colleagues | Reading in many languages | Everyday | Not in cards | | May 2024 | Nov 2026 | ||
| WeirdML v3 Agents solve 11 hand-made machine learning and data analysis tasks on a GPU with unfamiliar data and hidden test answers. Scored by token-weighted best results. | Håvard Tveit Ihle (Norwegian Defence Research Establishment) | Machine learning engineering | Expert | Not in cards | Claude Fable 5.2 43.1% | Sep 2026 | Nov 2026 | ||
| WeirdML v2 Replaced in Sep 2026 | Håvard Tveit Ihle | Not in cards | GPT-6 Astra Pro 93.6% | Jun 2025 | Sep 2026 | ||||
| LegalBench Models do 162 legal reasoning tasks written by legal professionals, covering six types of legal reasoning such as issue spotting and rule application. | Stanford University (Hazy Research) and collaborators | Law | Expert | Not in cards | Claude Fable 5 88.6% | Aug 2023 | Oct 2026 | ||
| ALE-Bench Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants. | Sakana AI and AtCoder | Competitive programming | Competition | Not in cards | GPT-6 Astra 3,164 | Jun 2025 | Nov 2026 | ||
| LisanBench Models build the longest chain of distinct English words from a given start word, each differing from the last by one letter. The score sums valid chain lengths over all start words. | Lisan al Gaib (scaling01, independent) | Puzzles and chess | Everyday | Not in cards | | May 2025 | Oct 2026 | ||
| TaxEval v2 Models answer expert-written tax questions. A judge grades answer correctness and stepwise reasoning against expert solutions. | Vals AI | Finance | Expert | Not in cards | | Sep 2026 | |||
| Longform Creative Writing (EQ-Bench) Models plan, revise and then write a short story or novella over eight 1,000-word turns from a minimal prompt. A judge model scores quality, and slop, repetition and degradation are tracked. | EQ-Bench (Sam Paech) | Creative writing | University | Not in cards | Claude Opus 5 86.3 | May 2025 | Oct 2026 | ||
| A-Bench Models answer multiple-choice questions about AI-generated images, covering both meaning and low-level visual quality, to test them as judges of image generators. | Q-Future (Nanyang Technological University and Shanghai Jiao Tong University) | General visual questions | Everyday | Not in cards | | Jun 2024 | Jun 2025 | ||
| SimpleBench Models answer over 200 multiple-choice trick questions on spatial, temporal, and social reasoning that ordinary people solve. Scored by average accuracy over five runs. | SimpleBench Team | Abstract puzzles and logic | Everyday | Not in cards | Claude Fable 5.2 94.9% | Oct 2024 | Nov 2026 | ||
| Extended NYT Connections Models solve New York Times Connections puzzles with up to four extra trick words added. Each puzzle scores by the square of the share of groups found. | Lech Mazur | Language reasoning | School | Not in cards | | Oct 2024 | Nov 2026 | ||
| Linguistic Knowledge Benchmark (Fab AI) Models answer multiple-choice and short-answer questions about Kiswahili and English grammar, phonics, and spelling. Three model judges vote on short answers. | Fab AI | Language reasoning | University | Not in cards | GPT-5.4 97.8% | Sep 2026 | Aug 2026 | ||
| Dutch Book Reviews Dataset Models classify the sentiment of Dutch book reviews from Hebban using ratings supplied by readers. | Benjamin van der Burgh and Suzan Verberne | Reading in many languages | Everyday | Not in cards | | Oct 2019 | Mar 2026 | ||
| LLM Agronomy Benchmark (Farmers Business Network edition) 2024-08 Models answer agronomy questions from six groups, including questions from Farmers Business Network. Each question carries equal weight in the overall score. | gbstox | Other professions | University | Not in cards | | Aug 2024 | Oct 2026 | ||
| LLM Agronomy Benchmark (June 2024) 2024-06 Replaced in Aug 2024 | Not in cards | GPT-4o 89.6% | Jun 2024 | Jun 2024 | |||||
| LLM Agronomy Benchmark (original) Replaced in Jun 2024 | Not in cards | GPT-4 Turbo 83.8% | Jan 2024 | Nov 2023 | |||||
| MERA Models complete Russian-language tasks with several answer formats. The hub port grades short answers from Part 1 of the Russian Unified State Exam. | MERA Evaluation team | Knowledge in many languages | University | Not in cards | Claude Opus 4.6 0.86 | Nov 2023 | Apr 2026 | ||
| Open LLM Leaderboard v2 Hugging Face ran open models on IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro and averaged normalized scores. | Hugging Face | Composite indices | University | Not in cards | | Jun 2024 | Mar 2025 | ||
| IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run. | Vals AI | Competitive programming | Competition | Not in cards | GPT-6 Astra 100% | Oct 2026 | Oct 2026 | ||
| IOI (Vals) v1 Replaced in Oct 2026 | Not in cards | Claude Opus 5 91.7% | Aug 2025 | Aug 2026 | |||||
| MedScribe Models write SOAP notes from clinical visit transcripts. Notes are graded against expert gold-standard rubrics for required content and section structure. | Vals AI (data partner Protege) | Medicine | Expert | Not in cards | Claude Opus 5.5 91.4% | Feb 2026 | Nov 2026 | ||
| MedCode Models read de-identified patient records and output primary and secondary ICD-10-CM diagnosis codes. Answers are checked against codes assigned by professional coders. | Vals AI | Medicine | Expert | Not in cards | Claude Opus 5 63.6% | Feb 2026 | Nov 2026 | ||
| EKI Eesti mäng Models choose answers to 1,000 questions from an Estonian quiz book. The questions cover subjects including history, culture, and geography. | Institute of the Estonian Language | Knowledge in many languages | Everyday | Not in cards | | Jun 2026 | Nov 2026 | ||
| EKI Propaganda Resistance Models answer 75 questions in Estonian, English, and Russian without web search. Claude Opus 4.5 grades how each answer handles misleading premises and propaganda. | Institute of the Estonian Language and Propastop | Deception and alignment | Everyday | Not in cards | Claude Opus 5 97.6 | Jun 2026 | Nov 2026 | ||
| Chess Puzzles Models find the single best next move in 100 chess puzzles that Epoch generated with a chess engine. Scored by the share solved. | Epoch AI | Puzzles and chess | Competition | Not in cards | Claude Fable 5.2 79% | Dec 2025 | Nov 2026 | ||
| AGI-Eval Community Instruction-Tuned Language Models Models answer Chinese and English questions across reasoning, knowledge, learning, interaction and instruction following. | AGI-Eval (Meituan) | Reasoning in many languages | School | Not in cards | GPT-5.5 0.86 | Mar 2025 | Jul 2026 | ||
| FoQA Models answer questions about passages from Faroese Wikipedia. A native speaker checked and corrected automatically generated answers. | Annika Simonsen and colleagues | Reading in many languages | School | Not in cards | | Feb 2025 | Mar 2026 | ||
| EKI Mälumäng Models answer 800 Estonian quiz questions without answer choices. Politics and geography each contribute 80 questions. GPT-5.4 nano judges answers that miss an exact match. | Institute of the Estonian Language | Knowledge in many languages | Everyday | Not in cards | | Aug 2026 | Nov 2026 | ||
| GermanQuAD Models extract answers to German questions from manually annotated Wikipedia passages. | Timo Möller and colleagues | Reading in many languages | School | Not in cards | | Apr 2021 | Mar 2026 | ||
| HeQ Models extract answers from Hebrew Wikipedia and Israeli technology reporting, or identify questions that the passage cannot answer. | Bar-Ilan University, Webiks and collaborators | Reading in many languages | School | Not in cards | | May 2023 | Apr 2026 | ||
| Open Hebrew LLM Leaderboard Models answer questions, resolve pronouns, classify sentiment and inference, translate passages, and answer trivia in Hebrew. | Israeli National Program for NLP, Dicta and Webiks | Knowledge in many languages | School | Not in cards | | Mar 2024 | Apr 2026 | ||
| Judgemark v4 The model under test judges fixed creative-writing samples against low and high anchor responses. The score is how well its ratings separate writer models. | EQ-Bench (Sam Paech) | Taste and style | Expert | Not in cards | Claude Opus 4.6 90.7 | May 2026 | Aug 2026 | ||
| Judgemark v2.1 Replaced in May 2026 | Not in cards | Claude Sonnet 4.6 94.1 | Jan 2025 | Mar 2026 | |||||
| MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. | Vals AI (data from Vontive) | Finance | Everyday | Not in cards | Claude Opus 5 72.1% | Sep 2026 | |||
| Mystery Game Puzzles 100 mid-game positions from an undisclosed puzzle variant of a well-known game. Model picks the single best move, checked against an answer key. | Epoch AI | Puzzles and chess | Competition | Not in cards | GPT-6.2 Astra 95% | Jul 2026 | Nov 2026 | ||
| MedQA Bias Evaluation Models answer medical exam questions before and after Vals adds racial information. Vals compares accuracy across patient groups. | Vals AI | Medicine | Expert | Not in cards | o1 96.5% | Jan 2025 | Mar 2026 | ||
| OCR-VQA Models answer questions about book covers by reading visible text. The benchmark compares answers with facts such as a book’s title or author. | Anand Mishra and colleagues | Documents, charts and text in images | School | Not in cards | Kimi-VL-A3B-Instruct 82% | Sep 2019 | Apr 2025 | ||
| Wer wird Millionär? LLM Benchmark Models play 45 games of the German version of Who Wants to Be a Millionaire, each with 15 questions of rising difficulty. Early questions lean on German wordplay and idioms. | ikiruneo (GitHub user) | Knowledge in many languages | Everyday | Not in cards | GPT-6 Astra 644444.0 | Sep 2025 | Sep 2026 | ||
| Miðeind Icelandic Case Declension Models decline Icelandic adjective and noun pairs in four cases and two numbers. The public test has 2,400 forms for 300 phrases. | Miðeind | Reasoning in many languages | School | Not in cards | GPT-5.5 98% | Nov 2023 | Aug 2026 | ||
| Miðeind Icelandic Grammatical Error Detection Models decide whether an Icelandic sentence contains a grammatical error. The test uses 100 correct sentences and 100 incorrect sentences from the Icelandic Error Corpus. | Miðeind | Reading in many languages | School | Not in cards | Claude Fable 5 91.5% | Dec 2023 | Aug 2026 | ||
| RuneBench Coding agents play RuneScape on an emulated game server by writing TypeScript against the rs-sdk library, with game-wiki notes for reference. Tasks train one skill or gather gold within 15 or 30 minutes. Skill tasks are scored by the peak XP rate in any 15-second window, and a verifier reads the final game state. | Websim (Max Bittker, Surya Narreddi, Rob Haisfield, Sean Lee) | Video games | Everyday | Not in cards | GPT-6.2 Astra 7.85 | Feb 2026 | Nov 2026 | ||
| Miðeind Icelandic ARC-Challenge Models answer multiple-choice science questions in an Icelandic translation of ARC-Challenge. Miðeind used Claude 3.5 Sonnet to translate the questions. | Miðeind | Reasoning in many languages | School | Not in cards | GPT-5.5 95.2% | Jun 2024 | Aug 2026 | ||
| EuroEval ERRNews Models summarize Estonian radio news transcripts. EuroEval adds stories from TalTech’s training split to make a 2,048-story test split and keeps 256 stories for validation. | EuroEval | Reading in many languages | School | Not in cards | | Aug 2025 | Nov 2026 | ||
| Open LLM Leaderboard v1 Language models answer tasks from the original Open LLM Leaderboard suite. | Hugging Face | Composite indices | University | Not in cards | | Apr 2023 | Jun 2024 | ||
| Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Vals AI | Composite work indices | Expert | Not in cards | Claude Opus 4.8 70.7% | May 2026 | May 2026 | ||
| Vals Multimodal Index 1.2 Replaced in May 2026 | Composite indices | Not in cards | Claude Fable 5 74.2% | May 2026 | Aug 2026 | ||||
| Vals Multimodal Index v1 Replaced in May 2026 | Not in cards | Claude Opus 4.7 71.1% | Oct 2025 | Apr 2026 | |||||
| APEX-1 Models complete expert-written work cases with source documents for four jobs: investment banking, consulting, big law and primary care. A judge model grades each response against rubrics. | Mercor | Tasks from real jobs | Expert | Not in cards | | Sep 2025 | Nov 2026 | ||
| EuroEval Models answer questions, summarize text, and perform language understanding tasks across European languages. EuroEval combines task rankings on its European leaderboard. | Dan Saattrup Smart and EuroEval contributors | Reading in many languages | School | Not in cards | GPT-6.2 Astra 1.17 | Aug 2021 | Nov 2026 | ||
| HELM Safety Stanford CRFM runs five safety benchmarks over six risk categories, such as violence, fraud and discrimination, and reports one leaderboard. | Stanford CRFM | Refusing harmful requests | Everyday | Not in cards | GPT-5 nano 98.6 | Nov 2024 | Nov 2025 | ||
| SEA-HELM Models are evaluated in Filipino, Indonesian, Tamil, Thai and Vietnamese across five pillars: NLP classics, LLM-specific tasks, linguistics, culture and safety. | AI Singapore + Stanford CRFM | Reasoning in many languages | School | Not in cards | GPT-6 Astra 79.9 | Feb 2025 | Nov 2026 | ||
| Pre-Flight Models answer questions about aviation regulations, dispatch, pilot procedures and airport operations. The evaluator compares each choice with the answer key. | Airside Labs | Other professions | Expert | Not in cards | GPT-5.6 Sol 84% | Mar 2025 | Sep 2026 | ||
| HELM Lite A lightweight HELM board that runs models with in-context learning on ten scenarios. They include NarrativeQA, NaturalQuestions, OpenbookQA, MMLU, GSM8K, MATH, LegalBench, MedQA and WMT 2014. | Stanford CRFM | Composite indices | University | Not in cards | GPT-4o 93.8 | Dec 2023 | Mar 2025 | ||
| Caliper Bench v3 Models continue prose passages and write dialogue in character. The graders score writing craft, character consistency, and errors across sessions lasting up to 50 turns. | Caliper Bench | Creative writing | School | Not in cards | | Sep 2026 | Nov 2026 | ||
| EuroEval Faroese Grammatical Error Correction Models rewrite a Faroese student’s sentence to correct its grammar. An exact match to the reference correction earns credit. | EuroEval, using sentence pairs from the University of the Faroe Islands and the University of Copenhagen | Reading in many languages | School | Not in cards | | Aug 2026 | Jan 2026 | ||
| NarrativeQA Models answer questions about books and movie scripts, either from a Wikipedia summary or from the full story. The questions need an understanding of the whole narrative rather than one matching passage. Answers are free text, scored against two reference answers with F1, ROUGE-L or BLEU. | DeepMind and University of Oxford (Kočiský et al.) | Reading comprehension | Everyday | Not in cards | GPT-4o 0.80 | Dec 2017 | Feb 2025 | ||
| CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. | Center for AI Safety | Composite indices | Research frontier | Not in cards | | Dec 2025 | Nov 2026 | ||
| Icelandic WinoGrande Models complete Icelandic sentences by choosing which referent fits the context. The evaluation uses an Icelandic adaptation of WinoGrande. | Vésteinn Snæbjarnarson and colleagues | Reasoning in many languages | Everyday | Not in cards | | Jun 2022 | Aug 2026 | ||
| CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. | Center for AI Safety | Composite indices | Research frontier | Not in cards | Claude Fable 5.2 8.40 | Dec 2025 | Nov 2026 | ||
| ParseBench-2 Parsers extract text, formatting, tables, charts, layout, and forms from 4,317 page cases across 2,645 documents. Rule checks score six dimensions and average them equally. | LlamaIndex | Documents, charts and text in images | Expert | Not in cards | Claude Opus 5.5 87% | Nov 2026 | Nov 2026 | ||
| ParseBench v1 Replaced in Nov 2026 | Not in cards | Claude Opus 5.5 78.8% | Apr 2026 | Nov 2026 | |||||
| TeleQnA Models answer multiple-choice questions about telecommunications terminology, research, and technical standards. | Ali Maatouk and colleagues | Expert-written questions | Expert | Not in cards | | Oct 2023 | Feb 2026 | ||
| TeleTables Models answer questions about tables from 3GPP specifications, both from memory and with the table available. | Anas Ezzakri and colleagues | Expert-written questions | Expert | Not in cards | | Jan 2026 | Feb 2026 | ||
| LongShOTBench Models answer single-turn and multi-turn questions about long videos using visual, spoken, and ambient audio evidence. Weighted rubrics award partial credit. | MBZUAI | Long videos | Everyday | Not in cards | | Dec 2025 | Apr 2026 | ||
| Cyber ECI Epoch estimates model capability from cybersecurity benchmark scores and places the result on the general index scale. | Epoch AI | Composite indices | Expert | Not in cards | Claude Opus 5.5 171.1 | Oct 2026 | |||
| FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. | FoodTruck Bench maintainers | Simulated companies | Everyday | Not in cards | | Nov 2026 | |||
| Phare Models are probed in English, French and Spanish for hallucination, harmful misguidance, stereotypes and jailbreaks. Hallucination tasks cover factuality, misinformation, debunking and tool use. | Giskard | Hallucination rates | Everyday | Not in cards | Claude Haiku 4.5 83.2% | Feb 2025 | Jun 2026 | ||
| AGI-Eval Community Image and Video Understanding Models interpret images and videos. The board reports visual question accuracy and separate comparison win rates. | AGI-Eval (Meituan) | General visual questions | Everyday | Not in cards | | Feb 2025 | Apr 2026 | ||
| Razavi-Bench Models explain what happens in analog circuits and why. A language model awards up to four points per answer against a reference and a rubric. | Zhishuai Zhang, Oscar Toomey, and Taekwang Jang | Electronics | University | Not in cards | | Nov 2026 | |||
| Dubesor Lichess Puzzle Rating Models solve sequences of moves in 500 rated Lichess puzzles. The organizer fits a performance rating against puzzle difficulty using only first attempts. | Dubesor | Puzzles and chess | School | Not in cards | | Oct 2026 | Nov 2026 | ||
| Miðeind Icelandic Case Government Models inflect a subject or construct a verb and object phrase in Icelandic. The verb determines the grammatical case. The evaluator checks exact agreement. | Miðeind | Reasoning in many languages | School | Not in cards | Claude Fable 5.2 100% | Sep 2026 | Nov 2026 | ||
| Miðeind Icelandic Grammar Correction Models correct an Icelandic sentence or return it unchanged. Half the sentences contain one error. The score measures exact agreement with the expected sentence. | Miðeind | Chat and instructions in many languages | School | Not in cards | Claude Fable 5.2 83.9% | Sep 2026 | Nov 2026 | ||
| Miðeind Icelandic Idioms Models supply a missing word in an Icelandic idiom and explain its meaning. A program checks the missing word. A language model grades the explanation. | Miðeind | Reasoning in many languages | Everyday | Not in cards | Claude Fable 5.2 85.7% | Sep 2026 | Nov 2026 | ||
| Miðeind Icelandic Popular Culture Models continue Icelandic song lyrics and identify the sources of familiar quotations. The two tasks measure word overlap and exact agreement, respectively. | Miðeind | Knowledge in many languages | Everyday | Not in cards | Claude Fable 5.2 39.5% | Sep 2026 | Nov 2026 | ||
| Miðeind Icelandic Terminology Models explain Icelandic knitting and crochet terms and name local species from English and Latin names. A model grades definitions. A program checks species names. | Miðeind | Knowledge in many languages | University | Not in cards | Claude Fable 5.2 78% | Sep 2026 | Nov 2026 | ||
| Open LMM Reasoning Leaderboard Models solve multimodal reasoning tasks. OpenCompass ranks the models using the component results. | OpenCompass | Composite indices | University | Not in cards | | Dec 2024 | Apr 2025 | ||
| HELM Capabilities Stanford CRFM runs language models on a curated set of scenarios, one per key capability, and averages the scores. Prompt-level results are public. | Stanford CRFM | Composite indices | University | Not in cards | GPT-5 mini 81.9 | Mar 2025 | Nov 2025 | ||
| Open Japanese LLM Leaderboard Models perform Japanese language tasks drawn from more than 20 datasets. | LLM-jp and Hugging Face | Knowledge in many languages | University | Not in cards | | Nov 2024 | May 2025 | ||
| RuLES 2.0 Models follow explicit rules while talking with a user. Each scenario checks automatically whether the model broke a rule. | Norman Mu and collaborators | Following precise instructions | Everyday | Not in cards | GPT-4 Turbo 9.54% | Mar 2024 | Feb 2024 | ||
| RuLES 1.0 Replaced in Mar 2024 | Not in cards | GPT-4 Turbo 9.54% | Nov 2023 | Feb 2024 | |||||
| Icelandic WikiQA Models answer 1,900 factual questions drawn from Icelandic Wikipedia. A language model grades each answer against a human-reviewed reference. | Miðeind | Knowledge in many languages | Everyday | Not in cards | Claude Fable 5.2 86.6% | Jul 2024 | Nov 2026 | ||
| LLM Confabulation Leaderboard Models answer misleading questions about recent articles whose answers are not in the text. The board ranks confabulation rate against non-response rate. | Lech Mazur | Faithful to sources | Everyday | Not in cards | GPT-5 10.3% | Oct 2024 | Aug 2025 | ||
| Miðeind Icelandic Generative Quality Models write paragraphs in Icelandic. Málfríður checks for grammar errors, and a dictionary lookup counts words absent from the Database of Modern Icelandic Inflection. | Miðeind | Chat and instructions in many languages | School | Not in cards | GPT-6.2 Sol 87.7% | Sep 2026 | Nov 2026 | ||
| Agent Island Models play multi-agent games that require cooperation, conflict, and persuasion. | Agent Island authors | Multi-agent games | Everyday | Not in cards | GPT-5.5 0.88 | May 2026 | Jun 2026 | ||
| Maze Bench Agents explore connected rooms in a 3D puzzle game. Each of the 100 gems contributes one percentage point to the score. | Jonathan Pappas and the MazeBench team | Puzzles and chess | Expert | Not in cards | GPT-6 Astra 24% | May 2026 | Nov 2026 | ||
| Nejumi LLM Leaderboard 4 Models solve Japanese language, reasoning, coding, and alignment tasks. The organizers weight individual categories when calculating the overall score. | Weights & Biases Japan | Reading in many languages | School | Not in cards | Aug 2025 | ||||
| Nejumi LLM Leaderboard 3 Replaced in Aug 2025 | Not in cards | o3 0.83 | Jul 2024 | Aug 2025 | |||||
| Nejumi LLM Leaderboard 2 Replaced in Jul 2024 | Not in cards | Dec 2023 | |||||||
| Nejumi LLM Leaderboard v1 Replaced in Dec 2023 | Not in cards | Jul 2023 | |||||||
| Arabic Cultural Value Alignment Models classify statements about Arabic culture as true or false. The evaluation covers 58 subject areas. | Freedom Intelligence and the AceGPT team | Knowledge in many languages | School | Not in cards | | Sep 2023 | Jan 2025 | ||
| BAAnchmark Models build a 3D sheep herding game from the same Slovak prompt. Marek Dubovsky plays each runnable game and ranks its controls, playability, and bugs. | Dubovsky Labs | Web and app front ends | University | Not in cards | Claude Sonnet 5.5 1.00 | Nov 2026 | |||
| Earth-Bench V2 Agents analyze prepared satellite observations. The corrected release measures final answer accuracy, tool selection, argument accuracy, and execution efficiency. | OpenDataLab and collaborators | Apps and MCP servers | University | Not in cards | Claude Opus 5.5 80% | Oct 2026 | Oct 2026 | ||
| Earth-Bench v1 Replaced in Oct 2026 | Not in cards | GPT-5 66% | Sep 2025 | Sep 2025 | |||||
| Fiction.LiveBench Models answer 36 questions about 30 Fiction.live stories that need inference about plot and characters. Each story is tested at several context lengths. | Fiction.live | Reasoning over long documents | Everyday | Not in cards | o3-pro 97.2% | Feb 2025 | Jan 2026 | ||
| Visual Reasoning Benchmark (Fab AI) Models choose which picture completes a pattern, matches another shape, or shows a folded figure. Questions come from school exams in Zambia and India. | Fab AI | Spatial reasoning and counting | School | Not in cards | | Sep 2025 | Aug 2026 | ||
| GeoBenchX v2 Agents use GIS tools to join data, measure spatial relationships, and draw maps. Three model judges compare their work with corrected reference solutions. | Varvara Krechetova and Denis Kochedykov | Apps and MCP servers | University | Not in cards | Claude Opus 5.5 84% | Jun 2025 | Oct 2026 | ||
| GeoBenchX v1 Replaced in Jun 2025 | Not in cards | GPT-4o 51.5% | Mar 2025 | Feb 2025 | |||||
| Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. | Surge AI | Composite work indices | Expert | Not in cards | | Aug 2026 | Nov 2026 | ||
| CAIS Vision Capabilities Index Averages model scores on five vision tests run by the Center for AI Safety: ERQA, MindCube, SpatialViz-Bench, IntPhys 2 and EnigmaEval. | Center for AI Safety | Composite indices | Research frontier | Not in cards | Claude Fable 5.2 84.0 | Dec 2025 | Nov 2026 | ||
| Lech Mazur Creative Writing Models write short stories that must use 10 assigned elements such as characters, objects, and motivations. Judge models compare matched stories pairwise. | Lech Mazur | Creative writing | University | Not in cards | GPT-6.2 Astra 4.30 | Jan 2025 | Nov 2026 | ||
| TaxCalcBench v2 2 (TY 2025) Models read W-2, 1099 and other PDFs and compute full federal and state returns line by line. A return counts only when every evaluated line is exact. | Column Tax | Finance | Expert | Not in cards | Claude Fable 5.2 86% | Jun 2026 | Nov 2026 | ||
| TaxCalcBench 1 (TY 2024) Replaced in Jun 2026 | Not in cards | GPT-5.4 62.8% | Jul 2025 | Mar 2026 | |||||
| VISTA Models answer 758 single-turn prompts about one image each, covering perception and reasoning. Three LLM judges grade answers with yes/no rubric questions. The main score is average task accuracy. | Scale AI | General visual questions | Expert | Not in cards | | Dec 2024 | Mar 2026 | ||
| CaseLaw v2 Models answer private questions over recent Canadian court decisions, such as finding key cases and answering across several documents. | Vals AI (with Jurisage, Together AI and Stanford CodeX) | Law | Expert | Not in cards | Grok 4.3 79.3% | Aug 2025 | Apr 2026 | ||
| FACTS Search V2 Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle. | Google DeepMind + Google Research + Kaggle | Short factual questions | University | Not in cards | GPT-6 Astra 89.9% | Jul 2026 | Oct 2026 | ||
| FACTS Search v1 Replaced in Jul 2026 | Not in cards | | Dec 2025 | May 2026 | |||||
| Instrumentation Universal Knowledge v1.6 refresh Models answer 60 technician questions and 97 specialist questions from the June samples. Three model judges score responses under higher output limits. | RelayForge | Tasks from real jobs | Expert | Not in cards | GPT-6 Luna 93.3% | Oct 2026 | Oct 2026 | ||
| Instrumentation Universal Knowledge v1.5 Replaced in Oct 2026 | Not in cards | Grok-3 mini 86.4% | Jun 2026 | Apr 2026 | |||||
| Instrumentation Universal Knowledge v1.3 Replaced in Jun 2026 | Not in cards | Claude Opus 4.7 93.3% | Jun 2026 | Apr 2026 | |||||
| Pencil Puzzle Bench Models solve 300 pencil puzzles of 20 varieties with unique solutions, either in one shot or agentically with step checks. Every board state can be verified against the rules. | Approximate Labs | Puzzles and chess | Expert | Not in cards | Claude Fable 5 97.6% | Mar 2026 | Jun 2026 | ||
| DataAnalyticsBench Agents query a synthetic data warehouse to answer business questions. A judge model checks answers against a rubric and an answer key. | Plotly and Mila | Running a business | University | Not in cards | GPT-6 Astra 43.0 | Oct 2026 | Nov 2026 | ||
| OpenLMM Spatial Leaderboard Models answer spatial reasoning tasks involving images. OpenCompass compares performance across the component evaluations. | OpenCompass | Composite indices | University | Not in cards | GPT-5 62.5 | Jun 2025 | Aug 2025 | ||
| Parallel Search Capability Leaderboard Language models answer research questions with and without search, and Parallel ranks their accuracy, search lift, cost, and time. | Parallel Web Systems | Composite indices | Expert | Not in cards | | Sep 2026 | Nov 2026 | ||
| PolicyBench No-tools estimate of US taxes and benefits for 100 households across 18 programs. Headline metric is answers within $1 of PolicyEngine-US. | PolicyEngine | Finance | Expert | Not in cards | GPT-6 Sol 95.2% | Jul 2026 | Nov 2026 | ||
| Swiss Smartvote 2023 party agreement Models answer 75 Swiss election questions about their own policy positions. Researchers compare their answers with political parties and candidates to measure agreement. | Smartvote and Jannis Vamvas | Bias, fairness and politics | Everyday | Not in cards | Claude 3.7 Sonnet 80.1% | Aug 2023 | Mar 2026 | ||
| ClawProBench Agents on stateful runtimes are scored with their traces, covering evidence gathering, runtime routing, safety boundaries and repeated runs in frozen workplace setups. | ClawProBench authors | General assistants | Expert | Not in cards | GLM-5.2 80% | Aug 2026 | Jul 2026 | ||
| Galileo Agent Leaderboard v2 Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality. | Galileo | Customer service | Everyday | Not in cards | GPT-4.1 62% | Jul 2025 | Aug 2025 | ||
| Galileo Agent Leaderboard v1 Replaced in Jul 2025 | Function calling | Not in cards | Claude 3.7 Sonnet 0.95 | Feb 2025 | Apr 2025 | ||||
| Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. | Vals AI | Other professions | Expert | Not in cards | Claude Opus 5 76.9% | Jun 2026 | Oct 2026 | ||
| AA-LCR Extended Models answer questions that need reasoning across about 100k tokens of documents. Mercor adds 89 new expert-written tasks to the 100 public ones and reports both scores. | Mercor | Reasoning over long documents | Expert | Not in cards | GPT-5.6 Sol 79.5% | Aug 2026 | Oct 2026 | ||
| DAXBench v2 Models write DAX, the formula language of Power BI, for 94 tasks on understanding, writing and reasoning over the Contoso sample model. Each answer runs inside real DAX queries in several report contexts, such as detail rows, subtotals, the grand total and slicers. Harder tasks earn more points, with bonuses for best practices and efficient code. | Maxim Anatsko, with SQLBI | Data science and SQL | University | Not in cards | GPT-6 Astra 95.9% | Oct 2026 | Nov 2026 | ||
| HealthBench-Psych v2 Models respond to 611 health conversations that clinicians selected for mental health relevance. Three model judges score each answer against its physician rubric. | MindBench.ai | Medicine | Expert | Not in cards | Claude Sonnet 5.5 0.66 | Sep 2026 | Sep 2026 | ||
| HealthBench-Psych v1 Replaced in Sep 2026 | Not in cards | Kimi K2.6 0.63 | Aug 2026 | Sep 2026 | |||||
| BRIDGE Models work on 87 tasks built from real clinical text, such as electronic health records, in nine languages. The tasks span triage, consultation, information extraction, diagnosis, prognosis and billing codes across 14 specialties. Each task uses its own automatic metric, and the board reports the average. | Harvard Medical School and Brigham and Women's Hospital (YLab) | Medicine | Expert | Not in cards | | Apr 2025 | Aug 2026 | ||
| PCT (Polarized Contrastive Pairs) Models answer matched left-coded and right-coded prompts, such as Obama and Reagan. Judges score whether sentiment and helpfulness stay consistent across each pair. | Center for AI Safety | Bias, fairness and politics | Everyday | Not in cards | | May 2026 | Nov 2026 | ||
| Ramp SWE-Bench v2 Agents solve 500 engineering tasks from Ramp's production backend. A panel of model judges compares each patch with the code engineers merged. | Ramp Labs | Fixing real repositories | Expert | Not in cards | GPT-6.2 Astra 76.4% | Nov 2026 | Nov 2026 | ||
| Ramp SWE-Bench v1 Replaced in Nov 2026 | Not in cards | Claude Fable 5 88.5% | Jun 2026 | Jul 2026 | |||||
| Spatial Agent Index v2 Agents analyze geospatial files and produce numbers, maps, or lists. Axis Spatial averages the scores for each included evaluation. | Axis Spatial | Apps and MCP servers | University | Not in cards | Claude Opus 5.5 86.0 | Nov 2026 | Oct 2026 | ||
| Spatial Agent Index v1 Replaced in Nov 2026 | Not in cards | Oct 2026 | |||||||
| BU Bench V2.1 Browser agents do 200 real web tasks. A findings judge checks tool evidence, answers and files, and deterministic integrity checks run as well. | Browser Use | Web browsing | Everyday | Not in cards | GPT-6.1 Sol 81.3 | Sep 2026 | Oct 2026 | ||
| BU Bench V2 V2 (60 tasks) Replaced in Sep 2026 | Not in cards | GPT-6 Astra 80.6 | Jan 2026 | Sep 2026 | |||||
| Misguided Attention Models get slight variations of famous riddles, paradoxes and thought experiments. A model fails when it answers the well-known original instead of the problem as stated. | cpldcpu (GitHub user) | Abstract puzzles and logic | Everyday | Not in cards | Grok-3 mini 73.7% | May 2024 | Jun 2025 | ||
| AraGen Models answer Arabic questions in four tasks: question answering, reasoning, orthographic and grammatical analysis, and safety. An LLM judge scores each answer on the 3C3H measure: correctness, completeness, conciseness, helpfulness, honesty and harmlessness. Only correct answers get credit on the other dimensions. | Inception and MBZUAI, with Hugging Face | Chat and instructions in many languages | Everyday | Not in cards | o1 84.3% | Dec 2024 | Apr 2026 | ||
| Hemingway-bench Professional writers compare two models' answers blind on real writing prompts, from bedtime stories to business documents. The comparisons become ratings. | Surge AI | Creative writing | Expert | Not in cards | Claude Fable 5 1,110 | Feb 2026 | Sep 2026 | ||
| QIMMA Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review. | Technology Innovation Institute (AlQadi, Alzubaidi et al.) | Knowledge in many languages | School | Not in cards | | Jan 2026 | Aug 2026 | ||
| SWE-bench Verified Extended Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores. | Mercor | Fixing real repositories | Expert | Not in cards | | Aug 2026 | Oct 2026 | ||
| Terminal-Bench 2.1 Extended Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores. | Mercor | Terminal tasks | Expert | Not in cards | | Aug 2026 | Oct 2026 | ||
| ContractorOS Quantity Takeoff Models fill two bills of quantities from electrical and plumbing drawings. Counts must match the estimator exactly. Measured lengths must fall within 5%. | ContractorOS | Tasks from real jobs | University | Not in cards | | Sep 2026 | Nov 2026 | ||
| GDPval Extended Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores. | Mercor | Tasks from real jobs | Expert | Not in cards | Claude Opus 5.5 7.10% | Aug 2026 | Oct 2026 | ||
| K-MetBench Models answer 1,774 questions from the Korean National Meteorological Engineer Examination under a zero-shot protocol. The benchmark reports answer accuracy across five subjects and separate visual and Korean-specific subsets. Expert rationales support an additional rubric score for reasoning. | KAIST, INEEJI and Kongju National University | Science questions | Expert | Not in cards | | Apr 2026 | Jul 2026 | ||
| SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. | Snorkel AI | Finance | Expert | Not in cards | Grok 4.7 30.5% | Oct 2026 | Oct 2026 | ||
| SnorkelUnderwrite 1.0 Replaced in Oct 2026 | Not in cards | GPT-5.4 91% | May 2025 | Mar 2026 | |||||
| Toloka Arena Enterprise Tool Use (November 2026) Agents complete private workflows across 11 industries. They must leave the right records and follow the required process in all five attempts. | Toloka | Customer service | Expert | Not in cards | | Nov 2026 | Nov 2026 | ||
| vCX Benchmark Models run inside a Retell voice agent on pass or fail checks for business rules, grounding, tool calls and speech-ready replies. Each suite reports its own pass rate. | Retell AI | Speech recognition and translation | Everyday | Not in cards | GPT-6.1 Sol 93.3% | Oct 2026 | |||
| BrowseComp Extended Agents browse the web to find hard-to-locate facts with short answers. Mercor adds 100 new expert-written tasks to the 130 public ones and reports both scores. | Mercor | Web browsing | Expert | Not in cards | Claude Sonnet 5.5 73% | Aug 2026 | Oct 2026 | ||
| BRUMO 2025 Models answer the 30 final-answer problems of the 2025 Brown University Math Olympiad. MathArena scores the average accuracy over four attempts. | Brown University (contest), MathArena (evaluation) | Competition problems with answers | Competition | Not in cards | | Apr 2025 | Feb 2026 | ||
| Envisioning Signals Benchmark Models identify early signs of change for an industry brief. Scores combine verifiable evidence, specific claims, recent sources, and coverage across the requested topics. | Envisioning | Running a business | University | Not in cards | Claude Opus 4.7 86.0 | May 2026 | Oct 2026 | ||
| KORA Simulated child users hold 3-turn conversations probing 26 child-specific risks. Judge model rates replies failing, adequate or exemplary. | KORA (non-profit) | Wellbeing and sycophancy | Not in cards | GPT-6 Astra 73.2% | Feb 2026 | Oct 2026 | |||
| Political Science LLM Benchmark 33 political-science text-coding tasks, 100 texts each (~3,300 items). Models scored by mean F1 against human-coded labels. | Hanno Hilbig (UC Davis) | Other | University | Not in cards | Claude Opus 5 71.4% | May 2026 | Oct 2026 | ||
| FlavourBench Powered v43 powered v43 Models choose ingredients for 640 tasks across four categories, including regional composition. Failed responses earn zero. Models must complete 608 tasks to qualify for ranking. | Josef Chen, Jakub Radzikowski, and Erim Hayretci | Taste and style | Everyday | Not in cards | Grok 4.6 73.4 | Aug 2026 | Aug 2026 | ||
| FlavourBench Complete Core complete core Replaced in Aug 2026 | Josef Chen and Erim Hayretci | Not in cards | Grok 4.6 65.1 | Aug 2026 | Aug 2026 | ||||
| Furniture Assembly Benchmark Model gets an IKEA manual and a photo of a partly built item, then names the faulty step or next step. Accuracy on 60 photos. | Epoch AI | Spatial reasoning and counting | Everyday | Not in cards | Claude Fable 5.2 90% | Sep 2026 | Nov 2026 | ||
| General-Bench Models perform multimodal understanding and generation tasks. General-Level averages task scores and measures performance above specialist baselines at progressively broader levels. | National University of Singapore, Nanyang Technological University, Zhejiang University, and collaborators | General visual questions | University | Not in cards | | May 2025 | Dec 2024 | ||
| LevelBench-100 v5.0 Models use ledger exports and supporting workbooks to close small business accounts. They reconcile transactions, apply accounting rules, and identify missing records. | Level | Finance | Expert | Not in cards | | Nov 2026 | Nov 2026 | ||
| LevelBench-100 v4.1 Replaced in Nov 2026 | Not in cards | GPT-6.1 Sol 81% | Oct 2026 | Oct 2026 | |||||
| MedXpertQA MM Extended Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores. | Mercor | Medicine | Expert | Not in cards | Claude Opus 5.5 71.2% | Aug 2026 | Oct 2026 | ||
| MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. | Epoch AI + METR | Building whole projects | Research frontier | Not in cards | Claude Sonnet 5.5 97.8% | Jun 2026 | Nov 2026 | ||
| MOMENT Eval Models use a calculator and Python to solve 91 structural engineering problems. They return a value or classification that a program checks against the answer key. | Thornton Tomasetti CORE studio | Tasks from real jobs | Expert | Not in cards | Claude Opus 4.7 95.6% | Aug 2026 | Jul 2026 | ||
| Open FlagEval VLM Leaderboard Models answer questions about images. The leaderboard compares their scores across visual understanding tasks. | Beijing Academy of Artificial Intelligence | General visual questions | University | Not in cards | | Feb 2025 | Apr 2025 | ||
| PAIRS (Parallel Images) Models answer questions about people in paired images. The evaluation tests whether changing demographic characteristics changes judgments about the subjects. | Kathleen C. Fraser and colleagues | Bias, fairness and politics | Everyday | Not in cards | GPT-4o 97.2% | Feb 2024 | Apr 2025 | ||
| Unicorn (Vision Safety) Models answer questions about objects in unusual images and sketches. The benchmark tests how well visual judgments hold up when the images change. | Haoqin Tu and colleagues | General visual questions | Everyday | Not in cards | Qwen2-VL-72B-Instruct 91.2% | Nov 2023 | Apr 2025 | ||
| VHELM Vision-language models are scored across nine aspects, including perception, reasoning, knowledge, bias, fairness, multilinguality, robustness, toxicity and safety, from many existing datasets. | Stanford CRFM | General visual questions | University | Not in cards | | Oct 2024 | Jun 2025 | ||
| Antidote Experts such as doctors, lawyers and engineers compare model answers to prompts from their own history. They judge which answer would still help them a month later. | Surge AI | Overall chat quality | Expert | Not in cards | Claude Fable 5 1,099 | May 2026 | Sep 2026 | ||
| BenCzechMark Models complete Czech language tasks spanning knowledge, comprehension, and reasoning. | Czech Language Community | Knowledge in many languages | University | Not in cards | | Oct 2024 | Aug 2026 | ||
| ClockBench Models read analog clock faces and answer questions about the time shown. Most clocks stay private to avoid training leakage. | Alek Safar | Spatial reasoning and counting | Everyday | Not in cards | Claude Fable 5.2 87.8% | Sep 2025 | Nov 2026 | ||
| KRYT Benchmark Models decide refund requests from transaction limits and event records. The grader checks policy compliance, valid actions, and consistency across rounds. | vaibhav lalwani | Customer service | Everyday | Not in cards | | Oct 2026 | Oct 2026 | ||
| Long-Horizon-Terminal-Bench 2.0 Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards. | Tencent HY + university collaborators (Zongxia Li et al.) | Long terminal sessions | Expert | Not in cards | GPT-6 Astra 69.2% | Oct 2026 | Sep 2026 | ||
| Long-Horizon-Terminal-Bench (LHTB) 1.0 Replaced in Oct 2026 | Not in cards | | Jul 2026 | Jul 2026 | |||||
| SMT 2025 Models answer 53 final-answer problems from the 2025 Stanford Math Tournament. MathArena scores the average accuracy over four attempts. | Stanford Math Tournament (contest), MathArena (evaluation) | Competition problems with answers | Competition | Not in cards | | May 2025 | Feb 2026 | ||
| TMMLU+ 1.1 Models answer traditional Chinese subject questions from a revised exam set. The revision corrects 197 answers and removes 539 questions from the original release. | iKala | Knowledge in many languages | University | Not in cards | Claude Opus 5 96.2% | Sep 2026 | Aug 2026 | ||
| TMMLU+ 1.0 Replaced in Sep 2026 | Not in cards | GPT-5 88.6% | Dec 2023 | Aug 2025 | |||||
| ArbMMLU-HT Models answer a human translation into Arabic of the 57 MMLU subjects. | Inception and MBZUAI (Jais project) | Knowledge in many languages | University | Not in cards | | Sep 2024 | Aug 2025 | ||
| Construction's Last Exam v2 Models measure construction drawings, answer questions across sheets, and price quantities under contract rules. The board reports geometry, drawing reasoning, and estimating separately. | Togal.AI | Tasks from real jobs | Expert | Not in cards | | Oct 2026 | Sep 2026 | ||
| Construction's Last Exam geometry-only Replaced in Oct 2026 | University | Not in cards | GPT-6.2 Astra 58.1% | Oct 2026 | Nov 2026 | ||||
| Open Ko-LLM Leaderboard 2 Models complete nine Korean evaluations covering reasoning, instructions, social values, safety, and other abilities. The organizers score submitted models on private test data. | Upstage and the National Information Society Agency | Reasoning in many languages | School | Not in cards | Qwen2.5-Coder-14B-Instruct 49.6% | Aug 2024 | Nov 2024 | ||
| Open Ko-LLM Leaderboard v1 Replaced in Aug 2024 | Knowledge in many languages | Not in cards | | Sep 2023 | Jun 2024 | ||||
| PlotPoints Round 4: Willingness and Judgment Models respond to scripted roleplay scenes. Two classifiers check whether models respect a stated limit and whether they refuse allowed requests. | Plotlight Studios | Deception and alignment | Everyday | Not in cards | Claude Fable 5.1 0.8 | Sep 2026 | Nov 2026 | ||
| Spatial Intelligence Index v2 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. | Axis Spatial | Composite indices | University | Not in cards | Claude Opus 5.5 88.7 | Nov 2026 | Oct 2026 | ||
| Spatial Intelligence Index v1 Replaced in Nov 2026 | Not in cards | Oct 2026 | |||||||
| ApprenticeBench Agent joins a simulated company and processes 100 accounts-payable bills in Odoo via GUI or API, learning from records and feedback. Cumulative success rate. | NeoCognition | Web browsing | University | Not in cards | Claude Opus 5.5 77% | Sep 2026 | Nov 2026 | ||
| Can AI close the books? FP&A reasoning Models complete synthetic financial close tasks, including accruals and reconciliations. The grader checks calculations and whether reviewer claims agree with the books. | Marco Polo | Finance | University | Not in cards | | Oct 2026 | Oct 2026 | ||
| CharXiv Extended Models answer descriptive and reasoning questions about scientific charts. Mercor adds 150 new expert-written tasks to the 5000 public ones and reports both scores. | Mercor | Documents, charts and text in images | University | Not in cards | | Aug 2026 | Oct 2026 | ||
| DAYJOB: Healthcare Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Medicine | Expert | Not in cards | Claude Opus 5.5 48.3% | Sep 2026 | Oct 2026 | ||
| GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. | Tencent ARC | Video games | Everyday | Not in cards | GPT-6 Astra 80.2 | Sep 2026 | Sep 2026 | ||
| MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. | Mercor | General visual questions | University | Not in cards | GPT-6.1 Sol 61.3% | Aug 2026 | Oct 2026 | ||
| MultiNRC Models answer over 1,000 reasoning questions written by native speakers in French, Spanish and Chinese. Questions cover linguistic reasoning, wordplay, cultural knowledge and culturally grounded math. A GPT-4.1 judge compares each answer with a short reference answer. | Scale AI | Reasoning in many languages | Expert | Not in cards | | Jul 2025 | Jul 2026 | ||
| Prompt to Product Agents build web applications from product briefs. Browser tests exercise features such as checkout arithmetic, saved filters, keyboard accessibility, and mobile layouts. | Shash | Web and app front ends | University | Not in cards | GPT-5.6 Sol 98.2% | Oct 2026 | Oct 2026 | ||
| SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. | Snorkel AI | Finance | Expert | Not in cards | Grok 4.6 25.1% | Oct 2026 | Oct 2026 | ||
| SnorkelFinance 1.0 Replaced in Oct 2026 | University | Not in cards | o3 81% | 2025 | Nov 2025 | ||||
| SWE Atlas Codebase QnA Extended Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores. | Mercor | Fixing real repositories | Expert | Not in cards | Claude Sonnet 5.5 44.7% | Aug 2026 | Oct 2026 | ||
| HiL-Bench Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary. | Scale AI | Fixing real repositories | Expert | Not in cards | | Apr 2026 | Nov 2026 | ||
| Last Translation Benchmark Models translate inputs that other systems struggled with. A model checks each translation against human-written rules. A translation passes only if it meets every rule. | Zouhar et al. | Translation | Research frontier | Not in cards | GPT-6 Astra Pro 45.3% | Sep 2026 | Oct 2026 | ||
| LLM Agronomy Benchmark Image Identification (four choices) Models choose among four labels for each photo of a crop disease, insect pest, or weed. One accuracy score combines the 150 questions. | gbstox | General visual questions | University | Not in cards | | Jun 2026 | Jun 2026 | ||
| MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. | Microsoft Research | Exam questions | University | Not in cards | GPT-4o 73.4% | Dec 2024 | Dec 2024 | ||
| Persian NLG Models summarize Persian texts, translate between Persian and two other languages, and write questions from passages. | MCINext | Translation | University | Not in cards | | May 2025 | Sep 2025 | ||
| RandomBench Models repeatedly choose an integer from 0 to 99. The benchmark measures how evenly they use the range and how often they return valid numbers. | Benjamin Brumfield | Abstract puzzles and logic | Everyday | Not in cards | Grok 4.6 0.42 | Nov 2026 | Nov 2026 | ||
| Vibe Code Bench 1-100 v1.1 Agents extend a working web application across successive requests while preserving earlier features. | Vals AI | Building whole projects | Expert | Not in cards | GPT-6.2 Astra 55.2% | Nov 2026 | Nov 2026 | ||
| Vibe Code Bench 1-100 Replaced in Nov 2026 | Not in cards | | Sep 2026 | Oct 2026 | |||||
| FINESSE-Bench Models answer financial certification questions and interpret trading indicators. The benchmark checks numerical and multiple-choice answers and uses a model judge for free responses. | Dmitry Stanishevskii and colleagues | Finance | University | Not in cards | GPT-5.5 0.90 | May 2026 | Jun 2026 | ||
| Itzi Roleplay Benchmark: Local Robustness Models play eight scripted episodes. Judges check whether each model remembers instructions, respects player decisions, and keeps characters and locations consistent. | Itzi | Creative writing | Everyday | Not in cards | MiMo-V2.5 100.0 | Sep 2026 | Sep 2026 | ||
| LongBench Pro Models answer English and Chinese questions over documents from 8,000 to 256,000 tokens. The benchmark applies task-specific metrics to 1,500 examples. | Ziyang Chen and colleagues | Reasoning over long documents | University | Not in cards | | Jan 2026 | Dec 2025 | ||
| MCS-Bench Models read seal inscriptions, identify calligraphy, and answer questions about Chinese paintings and artifacts. The overall score averages task scores. | South China University of Technology and INTSIG | General visual questions | Expert | Not in cards | | Feb 2025 | Mar 2025 | ||
| Persian IFEval Models respond to 457 Persian prompts while following constraints on wording, length, or structure. | MCINext | Chat and instructions in many languages | Everyday | Not in cards | o3 92.6% | May 2025 | Sep 2025 | ||
| The Aggregate Day-one Recall Check Models answer 34 questions about 12 public-domain books from memory. The scorer matches answers against a key and excludes empty replies. | Mikhail Doroshenko (The Aggregate) | Trivia and open questions | School | Not in cards | Claude Fable 5 61.8% | Aug 2026 | Nov 2026 | ||
| FinGREAT Models identify accounting relationships in financial passages. The authors compare 44 models against human annotations across 483 passages. | Carnegie Mellon University | Finance | University | Not in cards | Claude Fable 5.1 92.8% | Oct 2026 | Sep 2026 | ||
| GuiaCat Models classify the sentiment of restaurant reviews written in Catalan. | Projecte Aina | Reading in many languages | Everyday | Not in cards | | Oct 2022 | Jan 2026 | ||
| Memory Discipline Models answer project questions using saved memories. The tasks separate using relevant facts from asking for clarification when memories are absent or contradictory. | GiulioDER | Finding facts in long inputs | School | Not in cards | GPT-6 Astra 99.7% | Oct 2026 | Sep 2026 | ||
| MIZAN: A Persian LLM Leaderboard Models answer questions, complete sentences, follow instructions, and hold conversations in Persian. The leaderboard averages six task scores. | MCINext | Knowledge in many languages | School | Not in cards | o3 0.75 | May 2025 | Sep 2025 | ||
| Open Chinese LLM Leaderboard Models answer Chinese language tasks. The leaderboard compares model results across its evaluation suite. | Beijing Academy of Artificial Intelligence | Knowledge in many languages | University | Not in cards | | May 2024 | Jun 2024 | ||
| PerCoR Models read the beginning of a Persian passage and choose its original ending from four options. | MCINext | Reasoning in many languages | Everyday | Not in cards | | May 2025 | Sep 2025 | ||
| PerMMLU Models answer Persian multiple-choice questions from school curricula, university entrance exams, and Iranian cultural sources. | MCINext | Knowledge in many languages | School | Not in cards | | May 2025 | Sep 2025 | ||
| Persian MT-Bench Models follow Persian conversations across two to four turns. One asks for film directors, then actors, then the oldest and newest films. | MCINext | Chat and instructions in many languages | Everyday | Not in cards | o3 9 | May 2025 | Sep 2025 | ||
| PerspectiveGap Models assign information fragments to workers with different roles. A strict pass requires supplying all needed information without disclosing extra fragments. | Youran Sun and colleagues | Following precise instructions | University | Not in cards | GPT-5.5 62% | Jun 2026 | Sep 2026 | ||
| RiskRubric Evaluators score public models on six dimensions of risk. The scorecard combines the results into a letter grade. | Cloud Security Alliance, Haize Labs, Harmonic Security, and Noma Security | Deception and alignment | Expert | Not in cards | Claude Haiku 5.5 891 | Aug 2025 | Oct 2026 | ||
| AutoBench Agronomy LLM Benchmark Models answer questions from three types of farm operators and an agricultural researcher. Other models rank the answers. | AutoBench and EVJA | Tasks from real jobs | University | Not in cards | GPT-5.1 4.85 | Dec 2025 | Nov 2025 | ||
| AutoCodeBench Models solve automatically generated hard coding problems across many programming languages. A sandbox runs generated tests on each solution. | Tencent Hunyuan | Function-level problems | University | Not in cards | Claude Opus 4 52.4% | Aug 2025 | Aug 2025 | ||
| BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. | UCL DARK et al. | Video games | Everyday | Not in cards | GPT-6 Astra 68.3% | Nov 2024 | Sep 2026 | ||
| BaxBench Models write backend servers from a specification. Each solution is checked for correctness by tests and for security by end to end exploits. | ETH Zurich SRI Lab, LogicStar.ai, UC Berkeley and INSAIT | Secure code and review | Expert | Not in cards | Claude Opus 4.5 68.6% | Feb 2025 | Nov 2025 | ||
| EBR-bench Agent plays the board game Earthborne Rangers 10-30 times in a row. Scored on objectives completed (of 21) to test learning from experience. | Epoch AI | Puzzles and chess | Everyday | Not in cards | GPT-6.2 Astra 92.4% | Jul 2026 | Nov 2026 | ||
| ExtractBench Systems read a full business document and return a schema-valid JSON object with every requested value. Unified value F1 compares the output with the expected values. | LlamaIndex | Documents, charts and text in images | Everyday | Not in cards | GPT-6.1 Sol 95.3% | Jul 2026 | Oct 2026 | ||
| GDP.xlsx Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria. | Surge AI | Documents and spreadsheets | Expert | Not in cards | | Sep 2026 | Nov 2026 | ||
| GreekMMLU Models answer questions originally written in Greek for academic, professional, and government exams across 45 subjects. | Yang Zhang and colleagues | Knowledge in many languages | University | Not in cards | | Feb 2026 | Dec 2025 | ||
| Human Verification Gap Models respond to paired requests about the same synthetic image. The evaluation measures how the stated purpose changes their refusal rate. | Hersheyz | Refusing harmful requests | Everyday | Not in cards | Claude Sonnet 4.5 1.00 | Oct 2026 | Sep 2026 | ||
| HumanAgencyBench Models respond to 3,000 prompts about six ways assistants can support human agency, including asking clarifying questions and leaving important decisions to users. | Benjamin Sturgeon and colleagues | Deception and alignment | Everyday | Not in cards | Claude Sonnet 4.6 71.3% | Sep 2025 | Apr 2026 | ||
| IneqMath Models solve Olympiad-level inequality problems, recast as two checkable tasks: estimating a tight bound and predicting the relation between two expressions. IMO-level medalists wrote and reviewed the 200 test problems. Five model judges check the final answer and look for toy-case reasoning, logical gaps and numerical shortcuts, and a solution counts only if it passes all five. | Stanford University and collaborators | Written proofs | Competition | Not in cards | GPT-5 47% | Jun 2025 | Aug 2025 | ||
| LiveSecBench Models answer Chinese-language prompts that probe safety across areas such as legality, ethics, factuality and privacy. The paper's first edition turned a judge's pairwise comparisons into Elo ratings. The leaderboard's March 2026 edition gives each model an overall score from 0 to 100. The question bank is updated in dated editions. | Intoken (隐拓智安) and academic coauthors | Refusing harmful requests | Everyday | Not in cards | Claude Haiku 4.5 91.4 | Nov 2025 | Feb 2026 | ||
| Okapi Bench: API Freshness Models write integrations for changing APIs. The grader checks whether their calls still work with the current interfaces. | Okapi | Function-level problems | University | Not in cards | GPT-6 Astra 66.7% | Nov 2026 | Oct 2026 | ||
| OrangeSum Models summarize French news articles from Orange Actu. The dataset pairs articles with summaries written by journalists. | Moussa Kamal Eddine and colleagues | Reading in many languages | School | Not in cards | | Oct 2020 | Mar 2026 | ||
| PokéBench: Tool Assisted Benchmark Models play Pokémon Red with a screenshot, map, game text, and action shortcuts. They have 1,000 turns to reach ten milestones ending with Brock. | Cody Harris | Video games | Everyday | Not in cards | GPT-5.5 10.0 | Sep 2026 | Nov 2026 | ||
| Spiral-Bench v1.2 Models chat for 20 turns with a simulated suggestible user. A judge ensemble of Claude Sonnet 4.5, GPT-5 and Kimi-K2 logs protective and risky behaviors with intensity ratings, which combine into a safety score. | EQ-Bench (Sam Paech) | Wellbeing and sycophancy | Everyday | Not in cards | GPT-5 Chat 70.8 | Oct 2025 | Dec 2025 | ||
| Spiral-Bench v1.0 Replaced in Oct 2025 | Not in cards | GPT-5 87.0 | Aug 2025 | Sep 2025 | |||||
| Travel Agent Compassion Agents book travel activities when the customer does not mention animal welfare. The benchmark measures how often they choose a welfare-neutral option. | Compassion in Machine Learning | Deception and alignment | Everyday | Not in cards | Claude Opus 5 100% | Jun 2026 | Nov 2026 | ||
| TRLawBench Models answer 297 questions on Turkish law in three stages: 97 questions from official ÖSYM legal exams and 200 harder questions across 14 branches of law. Answers are checked against the answer key. A judge model also grades the reasoning from 0 to 10. | Beta Space Studio (Said Sürücü), with attorney Esat Erbil Tavus | Law | University | Not in cards | | Apr 2025 | Aug 2026 | ||
| ButtonBench Models resist repeated prompts that tempt them to press a forbidden button. The benchmark scores refusal and the creativity of the resistance. | Adam Holter | Deception and alignment | Everyday | Not in cards | o3-pro 97.6 | Dec 2025 | Dec 2025 | ||
| CMIMC 2025 Models answer 40 final-answer problems from the 2025 Carnegie Mellon Informatics and Mathematics Competition. MathArena scores the average accuracy over four attempts. | Carnegie Mellon University (contest), MathArena (evaluation) | Competition problems with answers | Competition | Not in cards | | May 2025 | Feb 2026 | ||
| Hebrew LLM Chat Leaderboard Models summarize documents, translate into Hebrew, answer trivia, resolve pronouns, and restore vowel marks. The suite reports each task separately. | Dicta and the Israeli National Program for NLP | Chat and instructions in many languages | School | Not in cards | | Nov 2025 | Aug 2026 | ||
| KorMedMCQA-Mixed Models take combined text and image exams. The 2022 set contains 281 questions and the 2023 set contains 307. Each year has a separate score. | Byungjin Choi, Seongsu Bae, Sunjun Kweon, and Edward Choi | Medicine | Expert | Not in cards | | Feb 2026 | Dec 2025 | ||
| KorMedMCQA-V Models answer Korean medical licensing questions using clinical images. The full test contains 1,534 questions and 2,043 images from 2012 through 2023. | Byungjin Choi, Seongsu Bae, Sunjun Kweon, and Edward Choi | Medicine | Expert | Not in cards | | Feb 2026 | Dec 2025 | ||
| OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported. | All Hands AI (OpenHands) | Fixing real repositories | Expert | Not in cards | Claude Fable 5 81% | Jan 2026 | Jun 2026 | ||
| Uncheatable Eval Models predict text from recently published documents. The evaluation measures perplexity across sources including news, research papers, fiction and code. | Kaifeng Tan and colleagues | Reading comprehension | University | Not in cards | | May 2024 | Apr 2026 | ||
| BabaIsBench Agents play levels of the puzzle game Baba Is You through a text interface and win by rewriting the rules. Models are compared on pass rate, speed and cost. | Quesma | Puzzles and chess | Expert | Not in cards | GPT-5.5 100% | Jun 2026 | Sep 2026 | ||
| Buyout Game Eight language models play a money-driven elimination game with private transfers and a buyout endgame. | Lech Mazur | Multi-agent games | Everyday | Not in cards | GPT-5.5 1975.4 | Jan 2026 | May 2026 | ||
| DualChem Models answer chemistry questions written by experts. The researchers report answer accuracy and assess whether models recognize possible misuse. | Scale AI with input from NIST CAISSI | Dangerous capabilities | University | Not in cards | GPT-5.6 Sol 67.3% | Oct 2026 | Oct 2026 | ||
| LLMzSzŁ Models answer Polish academic and professional exam questions drawn from the Central Examination Board archives. | Krzysztof Jassem and colleagues | Knowledge in many languages | University | Not in cards | | Jan 2025 | Apr 2025 | ||
| Open CoT Leaderboard Models answer logical reasoning questions with and without writing out their reasoning. | Logikon and Allen Institute for AI | Language reasoning | University | Not in cards | | Jan 2024 | Jan 2025 | ||
| SLR-Bench Models write Prolog rules that classify trains by their composition. The verifier runs each rule to determine whether its predictions are correct. | Lukas Helff and colleagues | Abstract puzzles and logic | University | Not in cards | GPT-5 81.8% | Jun 2025 | Apr 2026 | ||
| ToolComp Models chain several tool calls to answer 485 human-verified prompts. Most final answers are numbers. ToolComp-Enterprise gives 11 tools and ToolComp-Chat targets general chat tool use. Step labels also support process reward model tests. | Scale AI | Function calling | Expert | Not in cards | o1 70.1% | Jan 2025 | Mar 2025 | ||
| TxBench-Preclinical-Pharmacology Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision. | LatchBio | Genomics and life sciences | Expert | Not in cards | GPT-6.2 Astra 68% | Jun 2026 | Nov 2026 | ||
| WordleArenas Models guess a five-letter word from colored clues. Each arena changes how many clues give false feedback after a guess, from zero to five. | Chang Liu | Puzzles and chess | Everyday | Not in cards | Claude Opus 4 100% | Feb 2026 | Apr 2026 | ||
| Adversarial Humanities Benchmark Models respond to harmful requests rewritten in literary and scholarly styles. A judge measures how often the rewrites elicit unsafe answers. | Icaro Lab and Sapienza University of Rome | Jailbreaks and prompt injection | Everyday | Not in cards | Claude Sonnet 4.6 9.20% | Apr 2026 | Apr 2026 | ||
| ArxivRollBench Models reconstruct passages from recently published scientific papers. They order fragments, fill gaps and choose what comes next. | Zi Liang and colleagues | Reading comprehension | University | Not in cards | Kimi K2.6 70.8% | Jul 2025 | May 2026 | ||
| DeepsecBench Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings. | Vercel | Finding and exploiting real bugs | Expert | Not in cards | GPT-6.2 Astra 61.2 | Jul 2026 | Nov 2026 | ||
| HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. | Mercor | Expert-written questions | Expert | Not in cards | GPT-5.6 Sol 77.3% | Aug 2026 | Oct 2026 | ||
| INVALSI Italian LLM Benchmark Models answer Italian reading and grammar questions from 11 school tests. For questions with several parts, the model must answer every part correctly. | CRISP, University of Milano-Bicocca | Reading comprehension | School | Not in cards | Claude 3.5 Sonnet 92.2% | Jun 2024 | Mar 2025 | ||
| LEXam Models answer 7,537 questions from 340 law exams at Swiss, EU and international law courses, in English and German. Open answers are judged against reference answers, and multiple-choice items are exact match. | University of Zurich and collaborators | Law | Expert | Not in cards | GPT-5 70.2% | May 2025 | Dec 2025 | ||
| LLM Position Bias Benchmark Models judge pairs of responses twice, with the presentation order reversed. | Lech Mazur | Bias, fairness and politics | Everyday | Not in cards | MiMo-V2-Pro 19.8% | Jun 2026 | |||
| Decodability Assessment Benchmark Models flag words a child cannot yet sound out in a passage, given the phonics lessons the child has completed. The leaderboard reports adjusted balanced accuracy. | Fab AI | Language reasoning | University | Not in cards | GPT-5.5 95.9% | Sep 2026 | Jul 2026 | ||
| GSO-Bench Agents speed up code in 102 tasks across 10 real codebases, given a performance test. Scored by the share matching the expert commit's speedup. | GSO team (UC Berkeley et al.) | Speed and optimization | Expert | Not in cards | Claude Fable 5.1 88.2% | May 2025 | Sep 2026 | ||
| OncologyBench Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 50.7% | Oct 2026 | Oct 2026 | ||
| TxBench-Antibody-Discovery Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 67.7% | Aug 2026 | Nov 2026 | ||
| TxBench-Oligo-Discovery Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 62.2% | Sep 2026 | Nov 2026 | ||
| U-MATH Models solve 1,100 unpublished open-ended university math problems from teaching materials, balanced across six subjects. About 20% include images, and an LLM judge checks each solution. | Toloka AI and collaborators | Word problems | University | Not in cards | o1 86.8% | Dec 2024 | Jan 2025 | ||
| DualEntry Accounting AI Benchmark Models use accounting tools to look up accounts, reconcile balances, and draft entries for a test company. DualEntry checks their answers against fixed references. | DualEntry | Finance | University | Not in cards | | Oct 2026 | Nov 2026 | ||
| ElephantBench Models answer 1,094 closed-book questions about facts with multiple verified but conflicting accounts. A model judge assigns complete, partial, or failed recall. | Tencent Youtu Lab, Tsinghua University, and University of Warwick | Short factual questions | Expert | Not in cards | Kimi K3 52.4% | Aug 2026 | Jul 2026 | ||
| HEPToolBench 1.2 Models generate inputs for particle physics simulations and inspect the resulting files. The main suite has 28 tasks and a separate extension adds three debugging tasks. | Indian Institute of Science | Lab and data work | Expert | Not in cards | | Aug 2026 | Jun 2026 | ||
| KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. | Xueting Fang and colleagues | Medicine | Expert | Not in cards | Claude Opus 5 29.4% | Sep 2026 | Sep 2026 | ||
| LegalRewardBench Models choose which of two legal responses is better. The benchmark compares their choices with human-verified preferences across legal knowledge, analysis, decisions, and writing. | Harbin Institute of Technology and The University of Osaka | Law | Expert | Not in cards | Claude Opus 4.7 86.8% | Oct 2026 | Apr 2026 | ||
| MUSE Models answer questions about artworks used in language education. The tasks cover visual details, emotions, cultural context, and relationships within the image. | Luyao Zhu and colleagues | Other professions | School | Not in cards | GPT-5.6 Sol 56.7% | Sep 2026 | Jul 2026 | ||
| OpenGameEval Expanded Eval Set 87-eval set Agents extend existing Roblox games and build features from empty scenes. Roblox added 40 coding tasks to the original 47. | Roblox | Building whole projects | University | Not in cards | Claude Fable 5 50.3% | Apr 2026 | Jun 2026 | ||
| OpenGameEval 47-eval set Replaced in Apr 2026 | Not in cards | | Oct 2025 | Apr 2026 | |||||
| OpenVLM Video Leaderboard Models complete video understanding tasks. OpenCompass compares models across video evaluation datasets. | OpenCompass | Composite indices | University | Not in cards | | Sep 2024 | May 2025 | ||
| ReactBench Coding agents do 39 realistic React tasks in two tracks. Write React tasks come from merged pull requests and are checked by hidden behavior tests. Fix React tasks ask the agent to remove React performance, accessibility and quality issues. A task passes only if the tests pass and a pinned React Doctor scan finds no new graded issue. | Million (millionco) | Web and app front ends | Expert | Not in cards | Claude Opus 5.5 54.4% | Jul 2026 | Oct 2026 | ||
| VetEval Models answer examination questions written by licensed veterinarians. A fixed answer key measures clinical knowledge, and safety penalties reduce the overall index. | viggoVet | Medicine | University | Not in cards | Claude Opus 5.5 76.8 | Nov 2026 | |||
| xbench-ScienceQA Models answer postgraduate-level science questions written by experts, in Chinese, as part of the xbench AGI Tracking series. | xbench (HongShan) | Science questions | Expert | Not in cards | Claude Opus 4.8 76.4% | May 2025 | Sep 2026 | ||
| CircuitRubric Models write analog circuit netlists from design requests. A program compares transistor connections and relative device sizes with accepted reference circuits. | Manar El-Chammas | Electronics | University | Not in cards | Claude Opus 4.8 96% | Jun 2026 | Jun 2026 | ||
| DeepFinance Agents do private equity and investment banking work on 21 anonymized real deals: valuation, due diligence, underwriting and financial modeling. Each task comes with a data room of about 19 files and would take a professional about 27 hours. Rubrics of about 84 criteria grade the output, the workflow and the judgment behind each assumption. | Julius Research | Finance | Expert | Not in cards | Claude Fable 5.2 62% | Nov 2026 | Nov 2026 | ||
| Evalatro v2 Models play Balatro from the game state and try to clear Ante 12. Illegal moves reduce the score for progress through the game. | alesha-pro | Video games | Everyday | Not in cards | MiMo-V2.5-Pro 46.1 | Jun 2026 | Aug 2026 | ||
| HalachaBench Models answer 60 Hebrew rabbinical qualification exam questions. A model judge awards rubric credit on a scale from 0 to 100. | HalachaBench | Other professions | Expert | Not in cards | Claude Fable 5.2 85.8 | Nov 2026 | Nov 2026 | ||
| HELM Arabic Stanford CRFM runs models on 7 Arabic benchmarks in the HELM harness and ranks them by mean score. The tasks are AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and human-translated MMLU. | Stanford CRFM with Arabic.AI | Knowledge in many languages | School | Not in cards | Claude Opus 4.7 84.1 | Dec 2025 | Apr 2026 | ||
| K-Bench Models respond across multi-turn mental-health risk conversations involving suicide, self-harm, domestic violence, substance misuse, and no-risk cases. | Kivira Health and University of Roehampton | Wellbeing and sycophancy | Expert | Not in cards | GPT-5.5 99% | Sep 2026 | Jul 2026 | ||
| METR Capability Index METR fits an item response model to agent results on Time Horizon 1.1, TH2 and MirrorCode. The index places models on a common ability scale. | METR | Task length | Research frontier | Not in cards | GPT-6 Astra 859.0 | Nov 2026 | Sep 2026 | ||
| SORRY-Bench Models get unsafe requests across a fine-grained taxonomy of topics and linguistic variations. A judge model scores whether each answer refuses or complies. | Princeton University and collaborators | Refusing harmful requests | Everyday | Not in cards | Claude 2.1 5.23% | Jun 2024 | Sep 2024 | ||
| Vals RSI Index Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness. | Vals AI | AI research tasks | Research frontier | Not in cards | Claude Fable 5.2 0.38 | Aug 2026 | Nov 2026 | ||
| VERA-MH Suicide Risk Chatbots talk with simulated patients who show different levels of suicide risk. A judge grades the full conversation against a clinical rubric. | Spring Health and the VERA-MH Expert Council | Wellbeing and sycophancy | Expert | Not in cards | GPT-5.6 Terra 75.0 | Oct 2025 | Aug 2026 | ||
| AnalogCoder Models write Python programs that build analog circuits. A simulator checks whether each circuit performs its requested function. The authors report first-attempt success and how many tasks each model solves. | Yao Lai and colleagues | Electronics | University | Not in cards | | May 2024 | Mar 2025 | ||
| Contract Workflows (Models) Systems complete legal work involving contracts. The benchmark assesses whether each deliverable satisfies its legal and presentation criteria. | Legal Benchmarks | Law | Expert | Not in cards | Claude Fable 5.1 48.4% | Nov 2026 | |||
| Data Extraction (Models) Systems extract requested information from legal documents. The benchmark checks whether the resulting work satisfies the task criteria. | Legal Benchmarks | Law | Expert | Not in cards | Claude Opus 5.5 76.7% | Nov 2026 | |||
| DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. | Scale AI | Genomics and life sciences | Expert | Not in cards | | Jun 2026 | Nov 2026 | ||
| EpiBench Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 39.6% | Jun 2026 | Oct 2026 | ||
| Evalita-LLM Models perform tasks written originally in Italian. The suite includes admission exams, question answering and text generation. | Bernardo Magnini and colleagues | Reading in many languages | University | Not in cards | | Feb 2025 | Sep 2025 | ||
| Financial Coaching Bench Models interview a simulated client about a household financial problem and propose a plan. | Vals AI | Finance | Expert | Not in cards | Claude Opus 5.5 76.2% | Nov 2026 | Oct 2026 | ||
| HealthBench-Psych-Hard Models answer mental health questions from conversations that also appear in HealthBench-Hard. Three model judges use the original physician rubrics. | MindBench.ai | Medicine | Expert | Not in cards | Claude Sonnet 5.5 0.45 | Aug 2026 | Sep 2026 | ||
| InteractScience Models generate code for interactive scientific demonstrations. Evaluators test the program and assess its visual output. | OpenCompass and collaborators | Scientific code | University | Not in cards | Claude Sonnet 4 41.5 | Oct 2025 | Sep 2025 | ||
| NoveltyBench Models give several answers to the same open-ended request. The benchmark measures how many distinct responses they produce and how useful those responses are. | Yiming Zhang and colleagues | Creative writing | Everyday | Not in cards | Claude Fable 5.1 5.32 | Apr 2025 | Sep 2026 | ||
| RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. | Commure | Medicine | Expert | Not in cards | Claude Fable 5.1 94.2% | Nov 2026 | |||
| React Native Evals Models implement React Native features from task descriptions and fixed project contexts. A model judge checks the submitted files against explicit requirements, and evaluation aggregates requirement scores across tasks. | Callstack | Web and app front ends | University | Not in cards | Claude Fable 5.1 89% | Mar 2026 | Sep 2026 | ||
| RSNA Knee AUC10 Models label twelve knee findings from images, radiology reports, or both. The grader measures how well their predictions distinguish findings on ten reference studies. | Kirderf | Medicine | Expert | Not in cards | GPT-6.1 Sol 0.92 | Oct 2026 | Sep 2026 | ||
| scBench-Long Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 36.9% | Jun 2026 | Oct 2026 | ||
| SecIT Bench Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions. | Cribl AI Research Lab | Finding and exploiting real bugs | Expert | Not in cards | | Aug 2026 | Oct 2026 | ||
| Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior. | Snorkel AI | Building whole projects | Research frontier | Not in cards | GPT-6 Astra 47.6% | Oct 2026 | Oct 2026 | ||
| Snorkel Agentic Coding 1.0 Replaced in Oct 2026 | Expert | Not in cards | Claude Opus 4.6 65.2% | Jan 2026 | Feb 2026 | ||||
| SnorkelSpatial Models track particles on a 2D grid through 10 to 200 text-described moves, then report final positions and orientations. The 330 samples test allocentric and egocentric spatial reasoning. | Snorkel AI | Language reasoning | School | Not in cards | GPT-5.4 99% | Oct 2025 | Mar 2026 | ||
| SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 40.5% | May 2026 | Oct 2026 | ||
| VariantBench Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5 52.8% | Jul 2026 | Oct 2026 | ||
| VeriGen design problems Models complete Verilog modules for 17 digital circuits. Simulation checks whether the generated code satisfies each design’s testbench. | Shailja Thakur and colleagues | Electronics | University | Not in cards | | Dec 2022 | Jun 2026 | ||
| Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. | Arena (formerly LMArena) | Deception and alignment | Everyday | Not in cards | GPT-6.1 Sol 87.9 | Oct 2026 | Sep 2026 | ||
| BertaQA Models answer questions about Basque culture and general knowledge. Parallel Basque and English questions help distinguish language effects from familiarity with local topics. | Julen Etxaniz and colleagues | Trivia and open questions | Everyday | Not in cards | Claude 3 Opus 81.5% | Jun 2024 | Jun 2024 | ||
| DAYJOB: Accounting Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Finance | Expert | Not in cards | | Oct 2026 | Nov 2026 | ||
| EvoCode-Bench v2 Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit. | UniPat AI with Peking University, Tsinghua University and HKU | Building whole projects | Expert | Not in cards | Claude Opus 4.8 59.1% | Jun 2026 | Jun 2026 | ||
| EvoCode-Bench v1 Replaced in Jun 2026 | Not in cards | Claude Opus 4.7 54% | May 2026 | Apr 2026 | |||||
| GSM8K-V Vision-language models solve GSM8K word problems that are redrawn as multi-image visual scenes, so the facts must be read from pictures. | Zhejiang University | Visual math and diagrams | School | Not in cards | | Sep 2025 | Apr 2026 | ||
| InferenceBench Coding agents optimize a language model inference server within a fixed time budget. The harness measures serving speed and checks answer quality against a reference. | ELLIS Institute Tübingen and collaborators | Machine learning engineering | Expert | Not in cards | Claude Opus 5.5 12.1× | Nov 2026 | |||
| K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. | Chong Li and colleagues | Other professions | School | Not in cards | | Jun 2025 | Mar 2025 | ||
| KradleBench Models play live multi-agent Minecraft games and act by writing code. The overall score weights collaboration, economy and conflict at 30% each, and logic and creativity at 5% each. | Kradle | Multi-agent games | Everyday | Not in cards | GPT-6.2 Astra 78.3 | Nov 2026 | Nov 2026 | ||
| MetagenomicsBench Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 62.7% | Sep 2026 | Oct 2026 | ||
| OpenLMM Subjective Leaderboard Models generate open-ended answers to multimodal tasks. The board reports subjective evaluation results. | OpenCompass | Composite indices | University | Not in cards | GPT-4.1 76.6 | Apr 2025 | Apr 2025 | ||
| prinzbench Models research fact patterns under U.S. statutes and regulations and hunt for obscure public information online. 33 private questions run three times give a score out of 99. | prinz-ai | Law | Expert | Not in cards | GPT-5.6 Sol Pro 91.0 | Jan 2026 | Jul 2026 | ||
| SEAL Coding Models answer 1,000 private coding prompts across common languages and tasks. Human annotators rate adherence, correctness, efficiency and readability, then compare responses side by side to produce Elo-style scores. | Scale AI | Function-level problems | University | Not in cards | o1-mini 1,237 | May 2024 | Jan 2025 | ||
| Speko LLM Benchmark for Voice Agents v3 Models use simulated support tools to look up orders and issue refunds. The task score gives partial credit when a model stops early without causing harm. | Speko | Customer service | Everyday | Not in cards | | Nov 2026 | |||
| The Aggregate Trip Outcome Prediction Models read a packing list and itinerary, then guess what happened on a weekend trip. A judge scores four answers, and relative error scores the taxi fare. | Mikhail Doroshenko (The Aggregate) | Commonsense | Everyday | Not in cards | GPT-5.6 Sol 69.8% | Nov 2026 | Nov 2026 | ||
| 3LM Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation. | Technology Innovation Institute (Boussaha et al.) | Reasoning in many languages | School | Not in cards | | Jul 2025 | Apr 2025 | ||
| CADClamp 0.2 Models generate mechanical parts in seven CAD languages. Each part must meet its specification before the grader compares its printability with a reference part. | Christian Hill | CAD models | University | Not in cards | Claude Opus 5.5 0.93 | Sep 2026 | Sep 2026 | ||
| CADClamp 0.1 Replaced in Sep 2026 | Not in cards | Claude Opus 5 0.89 | Aug 2026 | Aug 2026 | |||||
| CVDP Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs. | NVIDIA | Electronics | Expert | Not in cards | Claude Opus 5.5 64.1% | Jun 2025 | Oct 2026 | ||
| DaLA Models distinguish acceptable Danish sentences from copies corrupted to resemble errors in real text. | Gianluca Barmina and colleagues | Reasoning in many languages | Everyday | Not in cards | | Dec 2025 | Feb 2026 | ||
| Deep20Bench 1.1 Models identify hidden people, characters, or objects by asking questions. The score averages the number of counted questions, with a fixed penalty for failed rounds, over three rounds for each of ten subjects. | Patrick Heusser and Markus Tuor | Puzzles and chess | Everyday | Not in cards | GPT-6 Astra 13.7 | Sep 2026 | Nov 2026 | ||
| EWoK Models decide which context makes a target statement plausible. Paired examples isolate concepts about the physical and social world. | Anna A. Ivanova and colleagues | Commonsense | Everyday | Not in cards | GPT-4 Turbo 91.2% | May 2024 | Oct 2024 | ||
| FailureSensorIQ Models answer multiple-choice questions on how industrial sensors relate to failure modes of assets. The Kaggle board shows it under IBM's AssetOpsBench citation. | IBM Research | Other professions | Expert | Not in cards | GPT-6 Astra 69.5% | Jun 2025 | Sep 2026 | ||
| IFEvalCode Models write code in eight programming languages under explicit constraints. The evaluator checks functional correctness and adherence to instructions separately. | Jian Yang and colleagues | Function-level problems | University | Not in cards | Claude 3.7 Sonnet 38.2% | Jul 2025 | Apr 2025 | ||
| Immigration Bench Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric. | Vals AI + Talent Mobility Fund (Renaissance Philanthropy) | Law | Expert | Not in cards | Claude Fable 5.1 75.4% | Oct 2026 | Nov 2026 | ||
| METR Time Horizon suite Software tasks with human completion times. Reports the task length at which a model succeeds 50% of the time. | METR | Task length | Research frontier | Not in cards | Claude Opus 4.5 4.1 h | Mar 2025 | Nov 2025 | ||
| MultiBLiMP Models compare minimally different sentences across languages. The benchmark tests whether the model assigns higher likelihood to the grammatical sentence. | Jaap Jumelet and colleagues | Reasoning in many languages | Everyday | Not in cards | | Apr 2025 | Apr 2025 | ||
| Real-SWE 1.1 Resolution rate on private enterprise coding tasks, averaged across five runs per task. | Specific Labs | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 56.8% | Oct 2026 | Oct 2026 | ||
| Real-SWE v1 Replaced in Oct 2026 | Not in cards | Claude Fable 5.1 38.8% | Sep 2026 | Sep 2026 | |||||
| SnorkelSequences Models compute values over integer sequences built from chains of filters, transforms and operators. The 250 procedurally generated questions test compositional multi-step math. | Snorkel AI | Word problems | School | Not in cards | GPT-5 77.6% | Jun 2025 | Mar 2026 | ||
| Surface Evolver Bench Agents write Surface Evolver datafiles that model liquid surfaces under physical constraints across 16 tasks. Deterministic checks give partial credit for matching hidden reference quantities. | Yann Henon | Research science | Expert | Not in cards | Claude Fable 5 95% | Jun 2026 | Sep 2026 | ||
| Swedish Benchmarks V1.1 Models answer Swedish language, document, and knowledge questions. The overall score combines normalized task performance and response time. | Drasko Markovic | Reading in many languages | School | Not in cards | | Apr 2026 | Aug 2026 | ||
| Swedish Benchmarks V1.0 Replaced in Apr 2026 | Not in cards | | Apr 2026 | Apr 2026 | |||||
| VisEditBench (TikZ diagram editing) Models edit TikZ code to follow instructions drawn on a diagram. A judge scores adherence, preservation of other content, and readability. | Tohoku University and collaborators | Spatial reasoning and counting | University | Not in cards | | Sep 2026 | Apr 2026 | ||
| Argo-Bench Agents query a simulated food delivery company's warehouse and act on their findings. The grader measures the business consequences of their decisions across 210 tasks. | TextQL | Simulated companies | Expert | Not in cards | Claude Fable 5.2 70.2 | Oct 2026 | Nov 2026 | ||
| Argus Inspection Models inspect images for small details that change the sensible response to a question. A judge scores whether the response identifies the relevant detail and chooses an appropriate action. The score combines those judgments and assigns zero when a separate description fails to recognize the detail. | Shanghai Artificial Intelligence Laboratory and university collaborators | General visual questions | Everyday | Not in cards | | Jun 2025 | May 2025 | ||
| AstroMLab 1 Models answer 4,425 multiple choice questions drawn from astronomy review articles. The authors publish the questions and grade submissions with a private answer key. | AstroMLab | Science questions | Expert | Not in cards | Claude 3 Opus 82.7% | Jul 2024 | Sep 2024 | ||
| BAZAAR Agents buy and sell goods in a simulated competitive market. | Lech Mazur | Multi-agent games | University | Not in cards | o3 7 | Sep 2025 | |||
| BizFinBench v2 Models answer expert financial questions grounded in business data. The score is average accuracy on eight offline tasks. | HiThink Research | Finance | University | Not in cards | GPT-5 61.5% | Jan 2026 | Dec 2025 | ||
| BizFinBench v1 Replaced in Jan 2026 | Not in cards | o3 73.9% | May 2025 | Apr 2025 | |||||
| Brokk Power Ranking Models solve 181 real tickets from a 1.1 million line Rust codebase. Model judges compare solutions in pairs, and a Bradley-Terry fit gives each model a strength score. | Brokk (SlopCop) | Fixing real repositories | Expert | Not in cards | | Aug 2025 | Nov 2026 | ||
| EusExams Models answer past public service examination questions in Basque. The questions come from several Basque institutions. | Julen Etxaniz and colleagues | Knowledge in many languages | University | Not in cards | | Mar 2024 | Apr 2025 | ||
| EusReading Models answer comprehension questions about Basque passages from past C1 proficiency examinations. The scorer checks their choices against the exam key. | Julen Etxaniz and colleagues | Reading in many languages | University | Not in cards | | Mar 2024 | Apr 2025 | ||
| EusTrivia Models answer trivia questions in Basque. The set combines elementary-school questions with harder general knowledge questions. | Julen Etxaniz and colleagues | Knowledge in many languages | School | Not in cards | | Mar 2024 | Apr 2025 | ||
| GBAEval Agents get 24 hours to write a Game Boy Advance emulator in Rust for WebAssembly. Replay, hardware, and audio tests compare it with Mesen2. | Mechanize | Building whole projects | Research frontier | Not in cards | Claude Opus 5.5 90.6% | May 2026 | Nov 2026 | ||
| GPBam Models write legal opinions on German public law cases and retrieve the statutes they need. Three judges grade essays against reference solutions. | Abdullah Al Zubaer and colleagues | Law | Expert | Not in cards | Claude Opus 5 76.4 | Nov 2026 | Aug 2026 | ||
| GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. | OpenCompass (Shanghai AI Laboratory) and collaborators | General assistants | Expert | Not in cards | GPT-5 11.4% | Apr 2026 | Dec 2025 | ||
| GTA v1 Replaced in Apr 2026 | Function calling | Everyday | Not in cards | GPT-4 Turbo 46.6% | Jul 2024 | May 2024 | |||
| MemLens Vision-language models and memory agents answer 789 questions over multimodal multi-session conversations, covering extraction, temporal reasoning, knowledge updates and refusal at four context lengths. | NVIDIA and collaborators | Finding facts in long inputs | Everyday | Not in cards | Kimi K2.5 53.6% | May 2026 | Mar 2026 | ||
| Open Tabletop GM Narrative Quality Probe v2 Models narrate twelve tabletop scenes, including player failure, deception, and changes in tone. Five judges score each scene for atmosphere, character portrayal, and game management. | Bobby Gray | Creative writing | Everyday | Not in cards | | Apr 2026 | Apr 2026 | ||
| Open Tabletop GM Narrative Quality Probe v1 Replaced in Apr 2026 | Not in cards | | Apr 2026 | Apr 2026 | |||||
| Paraphrase Stability Benchmark Models respond across a grid of paraphrases and contexts. The harness reports its Stability Agreement Average with a confidence interval. | Chauhan Balaji | Taste and style | School | Not in cards | GLM-5 1.00 | Oct 2026 | Sep 2026 | ||
| SnorkelGraph Models solve 200 procedurally generated graph problems, such as minimum vertex cover, given as node and edge lists in text. Answers are checked against verifiable ground truth. | Snorkel AI | Word problems | University | Not in cards | GPT-5.4 84.5% | Sep 2025 | Mar 2026 | ||
| ThaiExam Models answer multiple-choice questions from standardized examinations in Thailand, such as ONET, IC, TGAT, TPAT-1 and A-Level. | SCB 10X + Stanford CRFM | Knowledge in many languages | School | Not in cards | Claude 3.5 Sonnet 75.1% | Sep 2024 | Aug 2024 | ||
| The Aggregate Puzzle Portfolio Ladder Models solve 30 generated puzzles across five families. Each family has six questions with answers fixed by the generator. | Mikhail Doroshenko (The Aggregate) | Puzzles and chess | University | Not in cards | GPT-5.2 96.7% | Nov 2026 | |||
| WTP-Bench Models identify Pokémon from official artwork and silhouettes across 1,160 entries. The scorer checks exact forms, species, and evolutionary families. | Aaditya Baranwal, University of Central Florida | General visual questions | School | Not in cards | GPT-4.1 79% | Feb 2026 | Oct 2025 | ||
| AGI-Eval Community Base Language Models Base models answer Chinese and English questions across seven skill areas. The board reports objective accuracy. | AGI-Eval (Meituan) | Reasoning in many languages | School | Not in cards | Kimi K2 0.71 | Feb 2025 | Sep 2025 | ||
| ARCS Models identify ambiguities in a database question and write SQL for the intended interpretation. The scorer executes the query against the database. | Megagon Labs and collaborators | Data science and SQL | University | Not in cards | GPT-5 57.9% | Oct 2026 | Sep 2026 | ||
| CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. | Actava AI | Medicine | Expert | Not in cards | Claude Opus 5 54.7% | May 2026 | Jul 2026 | ||
| DELTA Models research Dutch legal questions and apply the law to facts. Lawyers judge accuracy and professional usefulness. | Legal Benchmarks and partner law firms | Law | Expert | Not in cards | Claude Fable 5.1 15.0 | Sep 2026 | Sep 2026 | ||
| FACTS Multimodal Models answer questions about input images and must combine what they see with world knowledge. A judge checks coverage and contradictions. | Google DeepMind + Google Research + Kaggle | Short factual questions | University | Not in cards | | Dec 2025 | Oct 2026 | ||
| Flash Fiction Stylistic Diversity Models write flash fiction from shared prompts. A reviewing model describes each story, and weighted Gower distance measures stylistic diversity within a writer’s work. | Lech Mazur | Taste and style | School | Not in cards | GPT-5 0.22 | Sep 2025 | Dec 2025 | ||
| General365 Models solve 365 hand-written reasoning problems and 1,095 variants in eight categories that need only K-12 knowledge. Scoring mixes rule-based and model-based checks. | Meituan LongCat Team | Abstract puzzles and logic | Everyday | Not in cards | | Apr 2026 | Feb 2026 | ||
| GeoGuesser Bench Agents explore offline Street View panoramas to report coordinates, name places, and find objects. Coordinate guesses earn distance points. Other answers require an exact match. | University of California, Berkeley | Spatial reasoning and counting | Everyday | Not in cards | | Nov 2026 | Jun 2026 | ||
| HELM Arabic Enterprise Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute. | Stanford CRFM with Arabic.AI | Knowledge in many languages | Expert | Not in cards | Claude Opus 4.7 77.9 | May 2026 | Apr 2026 | ||
| IDP Core Bench Models extract fields and tables, transcribe text, and answer questions about documents. The test covers about 2,000 invoices, receipts, forms, and handwritten documents. | Nanonets and IIT Indore | Documents, charts and text in images | Everyday | Not in cards | | Mar 2026 | |||
| lmgame-Bench Models play platformer, puzzle and narrative games such as Sokoban, Tetris, 2048 and Candy Crush through a Gym-style API with perception and memory scaffolds. | Hu et al. (lmgame-org) | Video games | Everyday | Not in cards | o3 87.7 | May 2025 | Aug 2025 | ||
| MOLTRON LLM Showdown v3.0 Models write JavaScript bots for a light-cycle game. Each working bot plays 600 matches against five reference opponents across six maps and both starting positions. | MOLTRON | Video games | University | Not in cards | GPT-6 Astra 69% | Sep 2026 | Sep 2026 | ||
| MOLTRON LLM Showdown v2.0 Replaced in Sep 2026 | Not in cards | Claude Opus 4.8 55.7% | Sep 2026 | May 2026 | |||||
| MOLTRON LLM Showdown v1.0 Replaced in Sep 2026 | Not in cards | GPT-5 73% | Sep 2026 | May 2026 | |||||
| ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. | Intern Science (Shanghai AI Laboratory) | Research science | Research frontier | Not in cards | | May 2026 | Aug 2026 | ||
| SecRespond Agents investigate compromised-host snapshots and write incident analyses and remediation plans. A model evaluator applies expert checklists separately to detection and response planning. | Alibaba NLP | Finding and exploiting real bugs | Expert | Not in cards | Claude Opus 4.7 79% | Jul 2026 | Jun 2026 | ||
| Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate. | Snorkel AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 39.3% | Jun 2026 | Sep 2026 | ||
| SoSBench Models answer hazardous requests drawn from real regulations in six scientific fields. Scored by how often the model complies with a request it should refuse. | SoSBench authors (Jiang et al.) | Hazardous knowledge | Expert | Not in cards | Claude Sonnet 4 10.6% | May 2025 | Aug 2025 | ||
| South Asian Music Understanding Models answer questions about Hindustani music theory and Bengali musical traditions. They also choose a continuation for a melody written in ABC notation. | University of Central Florida and Auburn University | Expert-written questions | University | Not in cards | | Jun 2026 | Mar 2025 | ||
| Terminal-Bench Pro Agents solve 400 terminal tasks across eight domains. Harbor distributes the 200 public tasks. Alibaba keeps the other 200 for private evaluation. | Alibaba | Terminal tasks | Expert | Not in cards | Claude Opus 4.5 51% | Dec 2025 | Feb 2026 | ||
| TRUE (Text Recognition and Understanding) Models recognize text in images and answer questions about its content. | BAAI FlagEval | Documents, charts and text in images | Everyday | Not in cards | | Mar 2025 | |||
| TxBench-Antibody-Development Agents use evidence to make manufacturing, quality, and clinical development decisions about candidate therapeutic antibodies. | LatchBio | Research science | Expert | Not in cards | | Nov 2026 | |||
| Verifying the Verifiers (Project APE CRED benchmark) Verifier LLMs must find planted errors (prose-table, paper-code, execution, data) in AI-generated economics papers. Credit needs a verbatim quote of the edit. | Project APE, University of Zurich (Willner, Yanagizawa-Drott) | Research science | Expert | Not in cards | GPT-6 Astra 99.5% | Jul 2026 | Sep 2026 | ||
| Access to Justice Benchmark Models interview a simulated client about a civil legal problem, explain their options, and direct them to local help. | Vals AI | Law | Expert | Not in cards | | Nov 2026 | Oct 2026 | ||
| AgriEval (Harbin Institute of Technology) Models answer Chinese agricultural exam questions across 29 subfields. The authors score 14,697 choice questions by accuracy and 2,167 open answers by ROUGE-L. | Harbin Institute of Technology and Institute for Advanced Algorithms Research | Exam questions | University | Not in cards | | May 2025 | Apr 2025 | ||
| BenchMAX Models follow instructions, write code, reason, use tools, translate, and retrieve information across 17 languages. Ten tasks use rule checks, model-judged wins, code tests, answer matching, or translation scores. | Nanjing University, Shanghai Artificial Intelligence Laboratory, University of Zurich, and Carnegie Mellon University | Reasoning in many languages | University | Not in cards | | Feb 2025 | Mar 2025 | ||
| ColorBench Models answer image questions about color perception, color reasoning, and robustness to color changes. The evaluator checks the selected answers against a single correct option. | University of Maryland | General visual questions | Everyday | Not in cards | | Apr 2025 | Feb 2025 | ||
| DatBench Models answer visual questions drawn from 33 datasets. The suite reports separate scores across nine visual capabilities. | DatologyAI | General visual questions | University | Not in cards | | Jan 2026 | Oct 2025 | ||
| DAYJOB: Law Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Law | Expert | Not in cards | GPT-6.2 Astra 41.4% | Oct 2026 | Nov 2026 | ||
| EvaOmni Models answer questions about videos and identify the audio and visual evidence behind their answers. The score accounts for both correctness and evidence timing. | Xiaoxing Hu and colleagues | Audio understanding and voice | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. | Snorkel AI | Finance | University | Not in cards | o3 81% | Mar 2026 | Nov 2025 | ||
| FrontierBreach Agents perform simulated enterprise tasks while processing untrusted material. The benchmark checks whether each agent resists instructions that conflict with its assigned task. | PromptArmor | Jailbreaks and prompt injection | Expert | Not in cards | | Nov 2026 | Nov 2026 | ||
| FrontierPhysics Agents do end-to-end physics research on 56 tasks that 42 physicists from 26 institutions built from their own work, across nine domains of theoretical, computational and experimental physics. Each task asks the agent to review the literature, plan, run the research and write a manuscript. Deterministic verifiers and expert rubrics grade whether the research is correct and publishable, over 3 trials. | BenchFlow, with physicists from UC Berkeley, Caltech and Harvard | Research science | Expert | Not in cards | Claude Opus 5.5 27.4% | Aug 2026 | Oct 2026 | ||
| Guesswork (The Aggregate) Models predict benchmark scores before publication. The maker ranks forecasts by mean absolute error after scaling each error by the variation in that benchmark’s scores. | Mikhail Doroshenko (The Aggregate) | Real-world events | University | Not in cards | GPT-6 Astra 0.56 | Jun 2026 | Sep 2026 | ||
| HWE-Bench v1.1 (RTL bug repair) Agents fix hardware bugs in 417 repository tasks. The revised tests check whether each fix restores the required behavior. | Peking University | Electronics | Expert | Not in cards | Claude Opus 5.5 99% | Nov 2026 | Oct 2026 | ||
| HWE-Bench v1.0 (RTL bug repair) Replaced in Nov 2026 | Not in cards | GPT-5.6 Sol 79.9% | Apr 2026 | Jul 2026 | |||||
| LLM-WikiRace Models play the Wikipedia game: starting from one article, they follow one hyperlink at a time to reach a target article. At each step the model sees only the links on the current page, and a game is lost after 30 steps. Three difficulty levels are set by the shortest path between the two pages. Scores are the share of games won at each level. | University of Oxford, University College London and University of Basel | Puzzles and chess | Everyday | Not in cards | | Feb 2026 | Feb 2026 | ||
| MIRAGE-MMST Models answer growers’ questions using photographs and written descriptions. They identify the cause of a problem and recommend how to manage it. | University of Illinois Urbana-Champaign and Amazon | General visual questions | University | Not in cards | GPT-4.1 68.0 | Jun 2025 | Apr 2025 | ||
| NeuroscienceBench Agents analyze neuroscience research data and report a scientific conclusion. The score counts correct conclusions among graded runs. | LatchBio | Research science | Expert | Not in cards | Claude Fable 5.2 41% | Nov 2026 | |||
| NoLiMa Models find a needle in a long haystack when the question and the needle share almost no words, so they must infer a latent link. Accuracy is compared across context lengths. | Adobe Research and LMU Munich (Modarressi et al.) | Finding facts in long inputs | Everyday | Not in cards | GPT-4.1 89.1% | Feb 2025 | May 2025 | ||
| o11y-bench Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment. | Grafana Labs | Terminal tasks | Expert | Not in cards | Claude Opus 5.5 93.8% | Apr 2026 | Oct 2026 | ||
| Open TTS Leaderboard 1.6.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. | Hugging Face | Audio understanding and voice | Everyday | Not in cards | | Nov 2026 | Jan 2026 | ||
| Open TTS Leaderboard 1.5.0 Replaced in Nov 2026 | Not in cards | | Nov 2026 | Jan 2026 | |||||
| Open TTS Leaderboard 1.4.0 Replaced in Nov 2026 | Not in cards | | Nov 2026 | Jan 2026 | |||||
| Open TTS Leaderboard 1.3.0 Replaced in Nov 2026 | Not in cards | | Oct 2026 | Jan 2026 | |||||
| Open TTS Leaderboard 1.2.0 Replaced in Oct 2026 | Not in cards | | Oct 2026 | Jan 2026 | |||||
| Open TTS Leaderboard 1.1.0 Replaced in Oct 2026 | Not in cards | | Oct 2026 | Jan 2026 | |||||
| Open TTS Leaderboard 1.0.0 Replaced in Oct 2026 | Not in cards | | Sep 2026 | Jan 2026 | |||||
| OpenDesign Arena Models create design artifacts in the OpenDesign tool, and the board reports an evaluation score out of 100 with cost and speed per artifact. | OpenDesign | Web and app front ends | Everyday | Not in cards | Kimi K3.1 Pro 91.0 | Nov 2026 | |||
| PatternEval Models answer 2,415 image prompts in thinking and non-thinking modes. A model judge flags exposed draft reasoning, repetition, contradictions, and unsupported reasoning. The Trigger rate measures responses with any flagged failure. | Tencent, Institute of Automation at the Chinese Academy of Sciences, and collaborators | Overall chat quality | Everyday | Not in cards | GPT-5.5 0.78% | Aug 2026 | Jun 2026 | ||
| ResearchCodeBench Agents read machine learning papers and fill missing Python code in the accompanying implementations. Unit tests check each of 212 snippets from 20 papers. | ResearchCodeBench team | Machine learning engineering | Expert | Not in cards | | May 2025 | Sep 2025 | ||
| RoboRewardBench Vision-language models act as reward models on real robot videos from Open X-Embodiment and RoboArena, judging task success and partial progress. Counterfactual relabeling adds calibrated failures and near misses. | Stanford (CRFM HELM, Percy Liang) + collaborators | Real robots | Everyday | Not in cards | GPT-5.5 0.63 | Jan 2026 | Apr 2026 | ||
| SlopCodeBench Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity. | University of Wisconsin-Madison (SprocketLab) | Building whole projects | Expert | Not in cards | GPT-5.6 Sol 32.1% | Mar 2026 | Sep 2026 | ||
| Structured Output Benchmark Models turn a text context into JSON that must match a given schema and hold the right values. The 5,324 records come from multi-hop questions, OCR text of documents and meeting transcripts, all given to the model as text. Seven automatic metrics, such as value accuracy, faithfulness and JSON validity, combine into an overall score that weights hard schemas more. | Interfaze (JigsawStack) | Following precise instructions | Everyday | Not in cards | GPT-5.4 87% | Apr 2026 | Jun 2026 | ||
| SWE-Together Coding agents work through 109 repository tasks rebuilt from real multi-turn sessions between users and coding agents. A simulated user replays the original user's goals and gives feedback when the agent needs it. Scores count tasks solved and the number of corrections the user had to make. | Meta | Fixing real repositories | Expert | Not in cards | Claude Fable 5.1 69.3% | Jun 2026 | Sep 2026 | ||
| TissueMicroscopyBench Agents examine tissue images and answer questions about cells, staining, and changes over time. | LatchBio | Research science | Expert | Not in cards | Claude Fable 5.2 34% | Nov 2026 | |||
| A.S.E 2.0 Models and agentic coding tools write code in real projects covering 29 CWE types. A mix of static analysis, tests and vulnerability proofs of concept checks security. | Tencent | Secure code and review | Expert | Not in cards | Nov 2025 | ||||
| A.S.E (AICGSecEval) 1.0 Replaced in Nov 2025 | Not in cards | Claude 3.7 Sonnet 63.0 | Jul 2025 | Aug 2025 | |||||
| Accounting Autocoding Models select a bookkeeping category for each expense. Ramp checks the first choice against the assigned category. | Ramp Labs | Finance | University | Not in cards | | Mar 2026 | Mar 2026 | ||
| BLiMP-NL Models compare Dutch sentence pairs that differ in grammatical acceptability. | Suijkerbuijk and colleagues | Reasoning in many languages | Everyday | Not in cards | | Jul 2025 | Dec 2025 | ||
| BrowseSafe++ v2 Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1. | Perplexity | Jailbreaks and prompt injection | University | Not in cards | GPT-6 Astra 91.5% | Oct 2026 | Sep 2026 | ||
| BrowseSafe-Bench v1 Replaced in Oct 2026 | Not in cards | Claude Sonnet 4.5 86.3% | Nov 2025 | Oct 2025 | |||||
| DataBench v1.1 Agents answer ambiguous business questions and make recommendations from a synthetic company warehouse. Model judges assess the reasoning, supporting evidence, artifacts, and final recommendation against task rubrics. | Hex | Running a business | Expert | Not in cards | Claude Opus 5.5 70.5% | Sep 2026 | Oct 2026 | ||
| Deep Research Bench (FutureSearch) Web research agents work on multi-step research tasks against RetroSearch, a frozen copy of 10,000 to 100,000 web pages per task. Answers are compared with answers worked out by people. | FutureSearch | Deep research | Expert | Not in cards | Claude Opus 4.6 0.55 | May 2025 | May 2026 | ||
| Dutch CoLA Models judge whether Dutch sentences drawn from descriptions of Dutch syntax are grammatically acceptable. | Computational Linguistics Groningen | Reasoning in many languages | Everyday | Not in cards | | Apr 2024 | Dec 2025 | ||
| EduBench-RU Models plan lessons, explain school subjects, and assist teachers in Russian and Chuvash. A panel of model judges scores responses against a teaching rubric. | Daniel Ivanov and Alexander Antonov | Other professions | University | Not in cards | | Mar 2026 | Mar 2026 | ||
| EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. | University of Illinois Urbana-Champaign + others | Simulated robots | Everyday | Not in cards | Claude 3.5 Sonnet (October 2024) 50.5% | Feb 2025 | Apr 2025 | ||
| Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. | Snorkel AI | Finance | Expert | Not in cards | Grok 4 53.1% | Jun 2025 | Mar 2026 | ||
| GSM8K Platinum Initial 268 Models answer the original 268-question cleaned GSM8K subset. The live matrix reports each model configuration separately. | MIT Madry Lab (Vendrow et al.) | Word problems | School | Not in cards | o1-preview 99.6% | Feb 2025 | Apr 2025 | ||
| LLM-Perf Leaderboard Language models generate tokens on a specified hardware and software configuration. The board reports latency, throughput, memory and energy alongside quality scores. | Hugging Face | Composite indices | University | Not in cards | | Jun 2024 | |||
| MathBench Models answer 3,709 math questions in five stages from basic arithmetic to college, in Chinese and English. Each stage also tests theory knowledge, and Circular Evaluation reshuffles answer options to check consistency. | OpenCompass (Shanghai AI Laboratory) | Word problems | University | Not in cards | GPT-4o 70.9% | May 2024 | Jun 2024 | ||
| MMCircuitEval Models answer questions about circuit diagrams, specifications, and design decisions. The authors combine text similarity with a language model’s judgment. | Chenchen Zhao and colleagues | Electronics | University | Not in cards | GPT-4V 69.4% | Jul 2025 | Mar 2026 | ||
| Physical-Engineering's Last Exam Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. | Google Cloud AI Research | CAD models | Expert | Not in cards | Claude Opus 5.5 0.49 | Oct 2026 | Nov 2026 | ||
| SATA-Bench Models select every correct option in 1,650 multiple-choice questions. Exact set match measures complete answers, with Jaccard overlap and option-count metrics as diagnostics. | SATA-Bench authors | Language reasoning | University | Not in cards | GPT-4.1 75.2% | Aug 2025 | |||
| SkateBench v2 Models name skateboard tricks from technical trick definitions. The score is the share named correctly. | Theo Browne (t3.gg) | Trivia and open questions | Everyday | Not in cards | | Mar 2026 | Mar 2026 | ||
| The Aggregate Long-tail Book Recall Models answer closed-book questions about 60 Project Gutenberg books. The scorer checks the answer text, and a second reviewer can accept equivalent wording. | Mikhail Doroshenko (The Aggregate) | Trivia and open questions | School | Not in cards | Claude Fable 5.2 77% | Nov 2026 | |||
| VIALS 2 2.0.0 Models interpret 161 professional life-science visual artifacts. The score is judged accuracy averaged across three rollouts. | Handshake AI Research | Genomics and life sciences | Expert | Not in cards | GPT-6 Astra 49.9% | Oct 2026 | Oct 2026 | ||
| VIALS 1.0.0 Replaced in Oct 2026 | Not in cards | Claude Opus 5 65.8% | Aug 2026 | Aug 2026 | |||||
| Vision2Web S3-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Z.ai (zai-org) | Web and app front ends | Expert | Not in cards | GPT-5.5 74.6% | Jun 2026 | Jun 2026 | ||
| Vision2Web S2-2026 Replaced in Jun 2026 | Not in cards | Claude Opus 4.6 43.5% | Mar 2026 | Mar 2026 | |||||
| Vision2Web S1-2026 Replaced in Mar 2026 | Not in cards | Claude Opus 4.5 52.7% | Mar 2026 | Dec 2025 | |||||
| AEC-Bench (Nomic) Agents search construction drawings and specifications to answer questions and identify conflicts. Task verifiers compare their outputs with known findings and award partial credit. | Nomic and coauthors | Tasks from real jobs | University | Not in cards | Claude Opus 5.5 70.2 | Mar 2026 | Oct 2026 | ||
| AILuminate Safety 1.2 Models answer 3,900 hidden English prompts, 325 in each of 12 hazard categories, picked with item response theory. A calibrated evaluator gives a 0 to 100 score. | MLCommons | Refusing harmful requests | Everyday | Not in cards | | Sep 2026 | Aug 2026 | ||
| MLCommons AILuminate 1.0 Replaced in Sep 2026 | Not in cards | | Dec 2024 | Aug 2026 | |||||
| MLCommons AI Safety Benchmark v0.5 Replaced in Dec 2024 | Not in cards | Apr 2024 | |||||||
| BIM-Edit Agents create, change, or delete elements in IFC building models. The evaluator checks the resulting geometry, attributes, and relationships. | University of Rostock and Clausthal University of Technology | CAD models | University | Not in cards | | Jun 2026 | Oct 2026 | ||
| BinaryAudit Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software. | Quesma | Finding and exploiting real bugs | Expert | Not in cards | Claude Opus 4.6 49.1% | Dec 2025 | Mar 2026 | ||
| BuzzBench Models analyze humor, such as why a joke or comic piece works. The leaderboard reports a score and response length per model. | EQ-Bench (Sam Paech) | Taste and style | Everyday | Not in cards | | Apr 2025 | |||
| CEQuest Models answer text questions about construction drawings, trade conventions, and quantity estimates. The original test has 101 multiple-choice questions and 63 true or false questions. | Florida International University and University of Florida | Tasks from real jobs | University | Not in cards | | Aug 2025 | Oct 2026 | ||
| CompileBench Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code. | Quesma | Terminal tasks | Expert | Not in cards | Claude Opus 4.1 100% | Aug 2025 | Oct 2025 | ||
| Construction AI Leaderboard Models answer construction questions, inspect drawings, edit building models, and generate validation rules. ONESTRUCTION averages the representative scores from five evaluations. | ONESTRUCTION | Composite indices | University | Not in cards | Claude Opus 5.5 73.7 | Nov 2026 | Oct 2026 | ||
| CulturalRiddles Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning. | Cohere Labs Community | Reasoning in many languages | Everyday | Not in cards | | Oct 2026 | Jul 2026 | ||
| Dictatorship Eval Scenarios at six difficulty levels testing whether models help with authoritarian requests, from blunt asks to requests hidden in code edits. LLM judge, no self-grading. | Free Systems (Andy Hall, Stanford Hoover) | Refusing harmful requests | Not in cards | Claude Fable 5 91.3% | Apr 2026 | Jul 2026 | |||
| IFC-Bench v2 Agents inspect IFC files and answer questions about building elements and their relationships. A model grades each answer against a reference. | Technical University of Munich and coauthors | CAD models | University | Not in cards | Claude Opus 5 65.8% | Feb 2026 | Oct 2026 | ||
| IFScale Models write business reports while including up to 500 specified keywords. The benchmark measures how many instructions they follow at each density. | Distyl AI | Following precise instructions | University | Not in cards | | Jul 2025 | Sep 2025 | ||
| Implicit Intelligence Agents respond to requests whose unstated requirements affect safety, privacy, or accessibility. Labelbox reports the share of scenarios each model passes. | Labelbox | General assistants | Everyday | Not in cards | Claude Opus 4.6 53.2% | Apr 2026 | |||
| Ishigaki-IDS-Bench Models convert building information requirements into IDS XML. An audit tool checks validity, and a scorer compares each requirement with the reference file. | ONESTRUCTION | CAD models | University | Not in cards | Claude Opus 5.5 63% | May 2026 | Oct 2026 | ||
| LLM Sycophancy Benchmark Models judge the same dispute from opposite narrators’ perspectives. Contradictory judgments count toward the sycophancy rate. | Lech Mazur | Deception and alignment | Everyday | Not in cards | Grok 4.5 0% | Jul 2026 | |||
| MA-Bench Models do military mission analysis under the joint planning process on long-context scenarios, graded against expert rubrics. | Scale AI (Lafuente-Mercado et al.) | Other | Expert | Not in cards | | Oct 2026 | Sep 2026 | ||
| MathCLUE Models solve problems from Chinese high school math competitions. | SuperCLUE | Competition problems with answers | Competition | Not in cards | | Apr 2025 | |||
| MecEng Models generate rigid and flexible multibody simulations from text. The evaluator checks system connections, geometry, and numerical outputs against engineering references. | University of Innsbruck and University of Augsburg | Scientific code | University | Not in cards | Claude Opus 4.8 91.4% | Aug 2026 | May 2026 | ||
| MobilePA-Bench Agents complete 1,705 mobile planning tasks using tools, persistent memory, reusable skills, and specialist agents. Task-specific checks evaluate tool calls, state changes, and interaction behavior. Additional gates check required memory access and skill loading. | MAI Team, Alibaba Token Hub, Alibaba Group | General assistants | Everyday | Not in cards | GPT-6 Astra 76.9% | Aug 2026 | Oct 2026 | ||
| Opus Magnum Bench Agents build machines on a hexagonal grid to make a target molecule. Each puzzle score compares the machine’s combined cost, cycles, and area with the human record. | Rob Haisfield, Max Bittker, and Surya Narreddi | Puzzles and chess | Competition | Not in cards | GPT-5.6 Sol 25.5 | Jun 2026 | Jul 2026 | ||
| RareDiseaseBench Agents combine clinical and biological evidence to diagnose and manage rare diseases. A rubric grades their diagnoses and plans. | LatchBio | Research science | Expert | Not in cards | | Oct 2026 | |||
| ROME Models solve reasoning problems selected to limit contamination. | BAAI FlagEval | Language reasoning | University | Not in cards | | Sep 2025 | Nov 2025 | ||
| RSI-Exam Agents get a weak method on a long research task and must turn it into one that scores better on hidden data. It measures recursive self-improvement on executable research. | AIMING Lab (UNC Chapel Hill) | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 53.6% | Aug 2026 | Nov 2026 | ||
| SpaceDG-Bench Models answer spatial questions under clean images and nine forms of visual degradation. Choice and binary answers use accuracy, while numerical answers use relative-error thresholds. The test has 1,102 verified questions from scenes excluded from the training set. | Shanghai Artificial Intelligence Laboratory and university collaborators | Spatial reasoning and counting | Everyday | Not in cards | | May 2026 | Apr 2026 | ||
| Spatial Pathfinding Reasoning Challenge Models trace paths across grids while satisfying rules about cells and edges. The scorer checks each coordinate sequence against valid solutions. | Lars Benedikt Kaesberg and colleagues | Puzzles and chess | School | Not in cards | GPT-6.1 Sol 83.6% | May 2025 | Sep 2026 | ||
| TasteVal 8 AI R&D tasks. Models iteratively design experiments. Taste measured as the compute needed to reach expert-level scores. | P-Zero Research | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 230.4× | Oct 2026 | Sep 2026 | ||
| The Aggregate Inversion and Execution Ladder Models predict the output of a short program or reconstruct the input to a digit transformation. The scorer checks exact answers across six difficulty levels. | Mikhail Doroshenko (The Aggregate) | Puzzles and chess | School | Not in cards | GPT-5.5 100% | Nov 2026 | |||
| TutorBench Models act as tutors on three tasks: adaptive explanations for a confused student, feedback on student work, and hints. Graded against sample-specific rubrics. | Scale AI | Other | University | Not in cards | | Oct 2025 | Apr 2026 | ||
| UnityBench-Mini3D Agents modify existing Unity projects. EditMode and PlayMode tests check the saved scene, component settings, and runtime behavior, with partial credit for completed milestones. | Unity China AI Hub and collaborators | Building whole projects | University | Not in cards | GPT-5.6 Sol 72.5% | Oct 2026 | Aug 2026 | ||
| VideoGAIA Agents inspect selected video segments, search the web, and read pages before answering questions. Model judges check answers, and the harness reports accuracy by category and difficulty. | CUHK, Ant Group, Tsinghua University and others | Video understanding | Everyday | Not in cards | | Aug 2026 | Aug 2026 | ||
| AGI-Pilled Bench Models forecast when AI will reach milestones that already happened in 2026. The score is the median years after 2026 across the forecast tasks. | Adam Holter | Forecasting | Everyday | Not in cards | Claude Fable 5.2 6.50 | Oct 2026 | Nov 2026 | ||
| Brainf**k coding benchmark Models write Brainfuck programs for tasks such as copying input and adding digits. Tests check the output of each program. | Pedro Schoen | Function-level problems | University | Not in cards | GPT-6 Astra 100% | Nov 2026 | Nov 2026 | ||
| CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. | Princeton University (Narasimhan and Liu labs) | Multi-agent games | Expert | Not in cards | Kimi K3 $22,148,357 | Jun 2026 | Aug 2026 | ||
| Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. | Naïve | Simulated companies | Expert | Not in cards | GPT-6 Astra $9,514 | Oct 2026 | Sep 2026 | ||
| Code2Video Benchmark 168 product-video briefs. Agent writes HTML motion-graphics code rendered by HyperFrames. A trained pairwise judge on five axes produces Elo. | HeyGen Research + Kaggle | Video generation | Expert | Not in cards | Claude Opus 5.5 1,585 | Sep 2026 | Sep 2026 | ||
| Creation-MMBench Models write creative pieces grounded in up to nine images plus a role, background and instructions, across 765 cases in 51 tasks. A judge scores quality and visual fact alignment against case-specific criteria. | OpenCompass (Shanghai AI Laboratory) | Creative writing | Everyday | Not in cards | | Mar 2025 | Feb 2025 | ||
| Databricks AI Index Coding agents complete tasks at different costs. The index compares cost per task and solution quality alongside enterprise usage statistics. | Databricks | Composite work indices | Expert | Not in cards | Claude Fable 5.1 96.0 | Nov 2026 | Nov 2026 | ||
| Doc2Harness Models build a coding-agent harness from an interface skeleton and a development document. The frozen harness then runs 328 scripted test cases in separate containers. | Institute of Computing Technology, Chinese Academy of Sciences | AI research tasks | Research frontier | Not in cards | GPT-6 Astra 53% | Oct 2026 | Sep 2026 | ||
| Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. | Andon Labs | Real robots | Expert | Not in cards | Claude Opus 5.5 97.3% | Jul 2026 | Sep 2026 | ||
| E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. | Qwen Team, Alibaba Group, with Taobao and Tmall Group | Running a business | University | Not in cards | GPT-5.6 Sol 14.3 | Aug 2026 | Aug 2026 | ||
| EvoEval Models solve Python problems that vary the constraints, interfaces, or logic of HumanEval tasks. Harbor ports the 100 problems in the Difficult subset. | Chunqiu Steven Xia and colleagues | Function-level problems | School | Not in cards | GPT-4 Turbo 68.6% | Mar 2024 | Apr 2024 | ||
| FrontierML Agents spend a fixed budget on GPUs and model APIs to train models for 21 tasks. An offline evaluator grades their submissions on held-out data. | Induction Labs | Machine learning engineering | Research frontier | Not in cards | Claude Opus 5.5 58.9 | Nov 2026 | Oct 2026 | ||
| GlobeBench Models act as text world models. Given an agent's past actions and observations, they predict the environment's next observation across code, enterprise, games, math and science, and dialogue. | Patronus AI | Interactive world models | Expert | Not in cards | GPT-6 Astra 49.6% | Oct 2026 | Sep 2026 | ||
| Hanabi Agent Benchmark Models play cooperative Hanabi in four-player teams. The benchmark scores shared board progress while hints and lives constrain coordination. | flatkey | Multi-agent games | Everyday | Not in cards | Claude Opus 5.5 20.5 | Oct 2026 | Oct 2026 | ||
| HW-Bench Agents complete six hardware engineering tasks for a 2.45 GHz antenna using native design and simulation tools. Code checks and calibrated model judges grade outputs against engineer-written criteria. The headline pass rate measures runs that satisfy every critical criterion, averaged across tasks. | Arena Physica | Electronics | Expert | Not in cards | GPT-6 Astra 67% | Nov 2026 | Nov 2026 | ||
| IndicParam Models answer multiple-choice questions from Indian competitive exams. The benchmark tests knowledge in low-resource Indic languages. | Ayush Maheshwari and colleagues | Knowledge in many languages | University | Not in cards | GPT-5 45% | Nov 2025 | Oct 2025 | ||
| Leipzig Benchmark Models answer 100 research-level math questions with known answers, written by 49 mathematicians at a Max Planck Institute workshop in April and May 2026. | Benchmarks in Leipzig workshop at MPI MiS (49 mathematicians), board on ScienceBench (Christian Stump) | Research mathematics | Research frontier | Not in cards | GPT-5.6 Sol Pro 91% | Jun 2026 | Sep 2026 | ||
| LiveNewsBench Release 2 Models search for answers to questions about recent news. The test limits searches and page visits and compares answers with curated references. | Yunfan Zhang, Kathleen McKeown and Smaranda Muresan | Questions that need search | Everyday | Not in cards | | Jan 2026 | Jan 2026 | ||
| LiveNewsBench Release 1 Replaced in Jan 2026 | Not in cards | GPT-5.2 21.5% | Sep 2025 | Dec 2025 | |||||
| LLM Public Goods Game Agents choose how many tokens to contribute to a common pool. The evaluation reports the fraction each model contributes across repeated games. | Lech Mazur | Multi-agent games | Everyday | Not in cards | | Feb 2025 | May 2025 | ||
| LLM Thematic Generalization Benchmark v2 Models rank eight candidates using three positive examples and three counterexamples. Version 2 filters ambiguous prompts and reports a harder subset of 703 items. | Lech Mazur | Language reasoning | University | Not in cards | Claude Opus 4.6 80.6 | Mar 2026 | Apr 2026 | ||
| MeasureBench Real-world Models read values and units from real instrument images. The score is the published overall measurement accuracy. | BAAI FlagEval team | General visual questions | Everyday | Not in cards | | Oct 2025 | Mar 2026 | ||
| MeasureBench Synthetic Models read values and units from generated instrument images. The score is the published overall measurement accuracy. | BAAI FlagEval team | General visual questions | Everyday | Not in cards | | Oct 2025 | Mar 2026 | ||
| MedSafe-Dx v0.3 Models read synthetic primary-care intake summaries and decide whether to flag a dangerous condition. Rule-based labels determine weighted error costs, with missed serious cases costing seven times a false escalation or incorrect danger label. The score normalizes these costs against a fixed escalation baseline. | Cortico Health Technologies | Medicine | Expert | Not in cards | Claude Opus 5.5 73.5 | Oct 2026 | Sep 2026 | ||
| MentalBench (Psychiatric Diagnosis) Models select psychiatric diagnoses for 24,750 synthetic cases. Exact matching requires both diagnoses when the case lacks the evidence needed to distinguish them. | KAIST and colleagues | Medicine | Expert | Not in cards | Claude Sonnet 4.5 62.7% | Feb 2026 | Nov 2025 | ||
| MetroLLM-Bench Agents operate a transit kiosk using tools for routes, fares, and service disruptions. The first score checks their actions automatically. The composite also includes model judgments. | Continker | Customer service | Everyday | Not in cards | | Jun 2026 | Aug 2026 | ||
| MindEval2 Models support simulated patients from MindSim, a tool-using harness with detailed profiles. Psychologist-written rubrics (MHQR) replace Likert scales for scoring. | Sword Health | Medicine | Expert | Not in cards | | Oct 2026 | Sep 2026 | ||
| MindEval v1 Replaced in Oct 2026 | Not in cards | Claude Opus 5.5 3.98 | Nov 2025 | Sep 2026 | |||||
| MM-HELIX Multimodal models solve 1,260 items from 42 synthetic tasks that need long reflective reasoning with backtracking. | MM-HELIX team (Zhao et al.) | Visual math and diagrams | University | Not in cards | GPT-5 84.5% | Oct 2025 | Aug 2025 | ||
| NorEval Models complete language understanding and generation tasks in both Bokmål and Nynorsk. The authors compare model performance with human baselines. | Vladislav Mikhailov and colleagues | Knowledge in many languages | School | Not in cards | | Apr 2025 | Apr 2026 | ||
| S-Eval Models respond to Chinese and English harmful prompts and attack variants. A trained safety judge measures the safe-response fraction and attack success rate. | Zhejiang University and Alibaba Group | Refusing harmful requests | Everyday | Not in cards | gpt-oss-120b 88.8% | May 2024 | Dec 2025 | ||
| SEAL Multilingual Models answer 1,000 private prompts per language, written by native speakers on general and cultural topics. Annotators rate each answer and compare pairs side by side, and boards rank models by Elo. | Scale AI | Chat and instructions in many languages | Everyday | Not in cards | o1 1,134 | May 2024 | Jan 2025 | ||
| SnorkelWordle Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate. | Snorkel AI | Puzzles and chess | Everyday | Not in cards | GPT-5 94% | Jul 2025 | Nov 2025 | ||
| TASTE 92 pairwise comparisons of AI safety research proposals. Model must match experienced researchers' preference. Accuracy versus ~77% human agreement. | Anthropic Fellows Program + Anthropic | AI research tasks | Expert | Not in cards | Claude Fable 5 60% | Aug 2026 | Jul 2026 | ||
| The Aggregate Lichess Pairwise Strength Models read two anonymized chess games and choose which designated player had the higher pregame Lichess rapid rating. Accuracy counts 25 choices. | Mikhail Doroshenko (The Aggregate) | Puzzles and chess | School | Not in cards | | Nov 2026 | |||
| The Aggregate Lichess Rating Prediction Models estimate both players' pregame Lichess rapid ratings from the moves of 25 anonymized games. The score is mean absolute error across 50 players. | Mikhail Doroshenko (The Aggregate) | Puzzles and chess | School | Not in cards | | Nov 2026 | |||
| TradeRank Arena Models choose positions in cryptocurrencies and US stocks using shared market data and simulated capital. The leaderboard ranks portfolios by return. | TradeRank.ai | Finance | University | Not in cards | Grok 4.7 0% | Jan 2026 | Nov 2026 | ||
| VerilogEval v2 Models write Verilog hardware descriptions from specifications. The hub port uses the specification-to-RTL tasks, where simulation checks generated designs against reference behavior. | NVIDIA and collaborators | Electronics | University | Not in cards | | Aug 2024 | Jun 2026 | ||
| Visual Aesthetic Benchmark Models see a set of images with the same subject and intent and must pick the best one, and in a harder setting both the best and the worst. The 400 sets hold 1,195 commissioned works across fine art, photography and illustration. Labels come from expert judgments, and only sets where experts largely agreed are kept. | BakeLab with University of Washington, UC Santa Barbara, Stanford, Notre Dame and IBM Research | Taste and style | Expert | Not in cards | Claude Sonnet 4.6 40.3% | May 2026 | Feb 2026 | ||
| VRBench Models answer multi-step reasoning questions about long narrative videos, which average 1.6 hours. The 1,010 videos come with 9,468 human-labeled question and answer pairs and 30,292 timestamped reasoning steps across seven reasoning types. Multiple-choice answers score the outcome, and a model judge scores the reasoning steps. | Shanghai AI Laboratory (OpenGVLab) and collaborators | Long videos | Everyday | Not in cards | Claude 3.7 Sonnet 68.2% | Jun 2025 | Jun 2025 | ||
| WebVR Models recreate a webpage from a screen recording and supplied assets. A model judge scores the rendered page against visual and interaction criteria. | StepFun and Tsinghua University | Web and app front ends | University | Not in cards | Kimi K2.5 79.1 | Mar 2026 | Feb 2026 | ||
| AbGen Models design ablation studies to test the contribution of a specified component. The benchmark compares the proposed experiments with expert annotations from 807 NLP papers. | Yale NLP Lab and collaborators | AI research tasks | Expert | Not in cards | | Jul 2025 | May 2025 | ||
| AgentBench FC Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format. | Tsinghua THUDM (Zhipu AI) | Function calling | Everyday | Not in cards | Claude Sonnet 4.5 58.9% | Oct 2025 | Sep 2025 | ||
| AgentBench v0.2 Replaced in Oct 2025 | Tsinghua THUDM | General assistants | Not in cards | GPT-4 4.01 | Aug 2023 | Feb 2024 | |||
| Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. | SecureBio | Hazardous knowledge | Research frontier | Not in cards | | Oct 2026 | |||
| AlgoTune Agents rewrite code for 154 math, physics, and computer science functions to run faster and stay correct. Scoring is harmonic mean speedup over reference code. | University of Tübingen and Princeton University | Speed and optimization | Expert | Not in cards | GPT-5.2 2.05× | Jul 2025 | Mar 2026 | ||
| AncientDoc Models transcribe historical Chinese pages, translate passages into modern Chinese, and answer questions about meaning, knowledge, and literary style. | Fudan University and ByteDance | Documents, charts and text in images | University | Not in cards | Qwen-VL-Max 5.77 | Aug 2025 | Apr 2025 | ||
| AutoBaxBench Models write backend servers for tasks that a pipeline generated. Tests and exploits are generated too. A larger task set built in the style of BaxBench. | ETH Zurich SRI Lab (von Arx et al.) | Secure code and review | Expert | Not in cards | Claude Fable 5.2 88.6% | Dec 2025 | Nov 2026 | ||
| Bedside Bench Models answer 500 physician-written clinical cases on diagnosis, workup, escalation and medication safety. An LLM judge grades each answer against the authors' criteria. | Doximity | Medicine | Expert | Not in cards | Claude Opus 5 94.6% | Oct 2026 | |||
| Biblically Informed Bot Learning Evaluation Models answer four-option questions about all 66 books of the Protestant Bible, biblical people, places, and measurements. The maker reports accuracy. | MushroomGecko | Trivia and open questions | School | Not in cards | | Apr 2025 | Mar 2026 | ||
| BoardBench 0.2 Models choose one move in each of 30 board positions across ten games. An engine grades each legal move and assigns zero to illegal moves. | BoardBench | Puzzles and chess | Everyday | Not in cards | GPT-6 Astra 67.2 | Oct 2026 | Oct 2026 | ||
| BoardBench 0.1 Replaced in Oct 2026 | Multi-agent games | Not in cards | Claude Fable 5 91.4% | Aug 2026 | Sep 2026 | ||||
| Chess LLM Benchmark (Ryan Brew) Models evaluate chess positions, choose a move, and explain the position. The primary move metric counts blunders against the Lc0 chess engine. | Ryan Brew | Puzzles and chess | School | Not in cards | | Apr 2026 | |||
| Collaborative Intelligence Benchmark A model works in a team with scripted teammates and live model partners, and a teammate creates a test: pressure to take an unsafe shortcut, misconduct to report, a need nobody states, a conflict, or a cheat that a weaker model left behind. The 240 tasks across 8 dimensions of teamwork come from real records, such as aviation accident reports and clinical cases, and from existing benchmarks, such as SWE-bench and negotiation datasets. Each task has a checkable outcome. A model is scored only on the tasks it solves alone, so the score measures teamwork, not ability. | Stanford University (James Zou's group) | Multi-agent games | Everyday | Not in cards | | Nov 2026 | Oct 2026 | ||
| ConStory-Bench Models write long stories, up to tens of thousands of words, from 2,000 prompts in four task types. An automated checker, ConStory-Checker, finds contradictions in facts, timelines, characters and world rules and cites the passages behind each one. Scores count consistency errors, using a taxonomy of 5 categories and 19 subtypes. | Microsoft and Singapore University of Technology and Design | Creative writing | Everyday | Not in cards | GPT-5 0.11 | Mar 2026 | Oct 2025 | ||
| ContractBench (Observation Contracts) Agents complete 33 API workflow tasks. Validators check whether agents preserve returned values exactly and use them while they remain valid. | SecurityLab at UC Davis | Function calling | University | Not in cards | Claude Opus 4.6 77.8% | May 2026 | Apr 2026 | ||
| CS-4k Models answer questions about the stages of computer science research using papers as evidence. A model judge scores answers on the held-out evaluation set. | Penghao Wang and colleagues | AI research tasks | Expert | Not in cards | o3 71.3 | Oct 2025 | Aug 2025 | ||
| dfbench Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately. | depthfirst | Finding and exploiting real bugs | Expert | Not in cards | Claude Mythos 5 69% | Aug 2026 | Sep 2026 | ||
| DrawingVQA Models answer 92 multiple-choice questions about 33 construction drawings. They read symbols, cross-reference details, and calculate quantities. | Louisiana State University, University of Illinois Urbana-Champaign, and an independent researcher | Documents, charts and text in images | University | Not in cards | | Apr 2026 | Nov 2025 | ||
| Flex-RSI Agents improve a starter solution and their own tools within a fixed budget. A hidden test split measures the final solution. | Flex-RSI team | Machine learning engineering | Expert | Not in cards | Claude Opus 5.5 1.52 | Oct 2026 | Oct 2026 | ||
| FretBench Models read guitar tablature in four tunings and name notes from string positions and fret numbers. The grader compares their answers with an answer key. | Jaiden Capra | Language reasoning | School | Not in cards | | Mar 2026 | Apr 2026 | ||
| GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result. | Carnegie Mellon University | Building whole projects | University | Not in cards | Claude Opus 5.5 70.6% | Feb 2026 | Sep 2026 | ||
| GradeBench Models read a student's handwritten answer to a university STEM exam question, with the rubric and a reference solution, and pick the rubric items to award. Scored by exact match with the items the instructor chose, averaged over five runs. | Pensive | Other professions | University | Not in cards | Claude Fable 5.2 57.3% | Nov 2026 | Nov 2026 | ||
| HWE Bench V2 (RISC-V CPU design) Agents improve a RISC-V processor from a shared starting design. The benchmark scores throughput on five hidden programs using clock speeds from Gowin's timing tool. | Felipe Sens Bonetto | Electronics | Expert | Not in cards | Claude Opus 5.5 7024.2 | Oct 2026 | Sep 2026 | ||
| HWE Bench V1 (RISC-V CPU design) Replaced in Oct 2026 | Not in cards | Claude Opus 5.5 983.2 | May 2026 | Sep 2026 | |||||
| Image2Struct Vision-language models turn images of webpages, LaTeX and musical scores into code that renders them. The render is compared with the input image. | Stanford CRFM | Documents, charts and text in images | University | Not in cards | GPT-4o 94.7 | Oct 2024 | Oct 2024 | ||
| LifeSci Diamond Models write regulatory affairs deliverables for drugs, biologics and medical devices, such as change assessment memos, dossier inconsistency logs and agency letters. The 55 tasks span five families: reporting categories, Module 2/3 dossier consistency, missing evidence, submissions and correspondence, and device vigilance. A model judge grades each answer against a task rubric, and a task passes only when every rubric line is met and no penalty fires. | Arca | Medicine | Expert | Not in cards | GPT-6 Astra 20% | Sep 2026 | Oct 2026 | ||
| LiveOCRVQA Models read stylized text in images and answer questions about it. | BAAI FlagEval | Documents, charts and text in images | Everyday | Not in cards | | Sep 2025 | |||
| LiveTradeBench Agents allocate simulated portfolios across US securities and Polymarket contracts using streamed prices and news. The researchers track portfolio returns over successive trading days. | U Lab at the University of Illinois Urbana-Champaign | Finance | University | Not in cards | Claude 3.7 Sonnet 12.1% | Aug 2025 | Sep 2025 | ||
| LLM Benchmark: Danske Idiomer Models match Danish idioms to pictures, complete expressions and write definitions. GPT-4.1 grades definitions, and answer keys cover the other tasks. | idiom.dk | Reading in many languages | Everyday | Not in cards | GPT-5.1 97.5% | Jan 2026 | Dec 2025 | ||
| LLM Divergent Thinking Creativity Benchmark Models generate sets of words that differ in meaning. | Lech Mazur | Creative writing | Everyday | Not in cards | o1-preview 4.79 | Mar 2025 | |||
| LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. | University of Illinois Urbana-Champaign | CAD models | Expert | Not in cards | GPT-6 Astra 66.5% | Oct 2026 | Sep 2026 | ||
| MacroscopeBench Models review commits from public repositories and must flag the known bugs a commit introduced without false reports. LLM judges match findings to the bugs. | Macroscope | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 83.9% | Nov 2026 | |||
| NaijaProverb-Bench Models select the intended meaning of Nigerian proverbs from four choices. One track adds the original Igbo or Hausa wording when available. | Jane Odum and Ayodele Dada | Reasoning in many languages | Everyday | Not in cards | | Apr 2026 | Apr 2026 | ||
| NanoGPT Speedrun Frontier Agents change a GPT optimizer on fixed GPUs. Each candidate must reach the target validation loss across eight fixed seeds using fewer training steps. | Prime Intellect | Machine learning engineering | Research frontier | Not in cards | Claude Fable 5 2726.0 | Aug 2026 | Aug 2026 | ||
| PokéBench: Raw Vision + Buttons Models play Pokémon Red from screenshots using eight Game Boy buttons. Game memory verifies ten milestones, with fewer turns breaking ties. | Cody Harris | Video games | Everyday | Not in cards | Claude Fable 5.1 10.0 | Oct 2026 | Nov 2026 | ||
| Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. | Ramp Labs | Finance | Expert | Not in cards | | Nov 2026 | Nov 2026 | ||
| SEAL Precise Instruction Following Models answer 1,054 private prompts with detailed instructions in nine categories. Human annotators check 12 yes or no criteria and compare responses side by side on a 7-point scale. | Scale AI | Following precise instructions | Everyday | Not in cards | o1 92.0 | May 2024 | Jan 2025 | ||
| SimulatorArena Models act as students or writing clients. Researchers compare their messages and assistant ratings with those from people who used the same assistants. | Georgia Institute of Technology and Microsoft | Multi-turn conversation | School | Not in cards | GPT-5 1.01 | Aug 2025 | |||
| SuperCLUE The CLUE team tests Chinese and foreign language models on periodic sets of Chinese questions covering language, professional knowledge, reasoning, agents and safety. Open answers are graded by a strong LLM judge, and results feed an overall ranking. | CLUE benchmark team (SuperCLUE) | Composite indices | University | Not in cards | | May 2023 | Nov 2026 | ||
| SuperCLUE-Terminal Agents complete tasks in a terminal. | SuperCLUE | Terminal tasks | Expert | Not in cards | MiMo-V2.6 Pro 64.7 | Sep 2026 | |||
| The Aggregate Chart Recall Models estimate ten years of monthly Wikipedia pageviews from each article title alone. The scorer averages each article’s root mean squared error in log views. | The Aggregate (Mikhail Doroshenko) | Trivia and open questions | School | Not in cards | GPT-6.2 Astra 0.22 | Nov 2026 | Nov 2026 | ||
| Thinkingbox-bench Agents complete 507 business workflows through tools and simulated user conversations. Executable checks require correct backend updates without forbidden side effects. | Microsoft and collaborators | Customer service | University | Not in cards | Claude Opus 5 66.5% | Aug 2026 | Sep 2026 | ||
| AEC-Bench (Theodore Galanos) Agents perform engineering calculations such as cable sizing and hydraulic design. Task verifiers score their outputs, and the leaderboard reports mean reward and completion. | Theodore Galanos | Tasks from real jobs | University | Not in cards | Grok 4.3 0.89 | May 2026 | Apr 2026 | ||
| AfroBench-Lite Models complete language understanding, question answering, reasoning, and generation tasks across 14 African languages. Seven datasets use task-specific metrics, including accuracy, F1, character-level translation scores, and exact match. | Masakhane, McGill University, Mila, University of Waterloo, University College London, Lelapa AI, and The African Research Collective | Knowledge in many languages | School | Not in cards | GPT-5 77.7 | Jun 2025 | Nov 2025 | ||
| Bio-VQA Models identify organisms, describe visible traits, judge spatial relationships, and recognize individual animals across photographs. The benchmark has 24 tasks. | Imageomics Institute and collaborators | General visual questions | University | Not in cards | | Aug 2026 | Jun 2026 | ||
| CORE-Bench Agents reproduce the results of published studies from their code and data. 270 tasks from 90 papers in computer science, social science and medicine come in three difficulty levels. | Princeton University | Research science | Expert | Not in cards | Claude Opus 4.5 77.8% | Sep 2024 | Nov 2025 | ||
| CorpBench Work Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly. | Revensi | Running a business | Everyday | Not in cards | GPT-5.6 Luna 92.7 | Sep 2026 | Sep 2026 | ||
| Crest Arcade Elemental Armory Models equip four elemental balls with cards, then select and aim a ball each round. The score averages match win rates across three levels. | Glimmer Index | Video games | Everyday | Not in cards | GPT-6.1 Terra 67.3% | Nov 2026 | |||
| Crest Arcade FreeCell Models move cards to the four foundations using a variable number of free cells. The score averages win rates across five tiers. | Glimmer Index | Puzzles and chess | Everyday | Not in cards | Claude Haiku 5.5 63.3% | Nov 2026 | |||
| Crest Arcade Hustle Sim Models buy upgrades and hire helpers to reach earnings targets in an idle game. The score compares simulated completion times with reference strategies. | Glimmer Index | Video games | Everyday | Not in cards | GPT-6.2 Astra 83.7 | Nov 2026 | |||
| Crest Arcade Micro Mouse Models steer a simulated robot through six daily mazes. The score compares each finish time with the median human time and gives unfinished mazes zero. | Glimmer Index | Video games | Everyday | Not in cards | | Nov 2026 | |||
| Crest Arcade Mines Models reveal or flag cells on Minesweeper boards that can be solved without guessing. The score averages win rates across four board sizes. | Glimmer Index | Puzzles and chess | Everyday | Not in cards | | Nov 2026 | |||
| Crest Arcade Sudoku Models fill complete Sudoku grids and report confidence. The score averages the share solved across seven equally weighted difficulty tiers. | Glimmer Index | Puzzles and chess | Everyday | Not in cards | Claude Opus 5.5 100% | Nov 2026 | |||
| Deta Studio AI Construction Takeoff Models use Deta’s tools to mark floor areas and count openings and electrical devices on four drawings. The rating combines accuracy, cost, and time. | Deta Studio | Tasks from real jobs | University | Not in cards | GPT-6.2 Sol 76 | Nov 2026 | Nov 2026 | ||
| DrafterBench Agents call drawing revision tools from written instructions. The evaluator records their operations and arguments across 1,920 tasks covering 12 types of revision. | Yinsheng Li, Zhen Dong, and Yi Shao | Function calling | University | Not in cards | Claude Sonnet 4.5 84.5% | Apr 2025 | Mar 2026 | ||
| Dutch Book Bench Models assign probabilities to related events in separate conversations. The grader checks whether inconsistent probabilities let a bettor guarantee a profit. | Yi Yu | Real-world events | University | Not in cards | | Oct 2026 | Oct 2026 | ||
| EarthVerse Agents investigate documented natural hazards using event data from different sources. The evaluator checks individual claims and the calculations that support them across 405 tasks. | Zhiqing Cui and colleagues | Lab and data work | Expert | Not in cards | Claude Fable 5 85% | Aug 2026 | Jul 2026 | ||
| Echo General AI Prediction Leaderboard Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models. | UniPat AI and Peking University | Real-world events | Expert | Not in cards | | Mar 2026 | Apr 2026 | ||
| EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. | University of Illinois Urbana-Champaign + others | Simulated robots | Everyday | Not in cards | Claude 3.7 Sonnet 67.7% | Feb 2025 | Apr 2025 | ||
| EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. | University of Illinois Urbana-Champaign + others | Simulated robots | Everyday | Not in cards | Claude 3.7 Sonnet 58.7% | Feb 2025 | Apr 2025 | ||
| EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. | University of Illinois Urbana-Champaign + others | Simulated robots | Everyday | Not in cards | Claude 3.7 Sonnet 28.5% | Feb 2025 | Apr 2025 | ||
| EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. | University of Illinois Urbana-Champaign + others | Simulated robots | Everyday | Not in cards | | Feb 2025 | Apr 2025 | ||
| EmbodiedVerse-Bench Models answer questions about spatial relationships and actions from visual observations. The leaderboard compares models across embodied reasoning tasks. | Beijing Academy of Artificial Intelligence | Embodied reasoning | Everyday | Not in cards | o4-mini 50.7 | Sep 2025 | |||
| FrogsGame post-training task Agent has 20 hours to post-train Qwen3-8B to solve FrogsGame puzzle boards. A hidden verifier scores the best checkpoint on unseen boards. | Thoughtful Lab (task in FrontierSWE v1) | Machine learning engineering | Research frontier | Not in cards | Claude Fable 5 34% | Apr 2026 | Aug 2026 | ||
| HELM Classic Models are evaluated with standardized prompts on the scenarios from the original HELM paper. Each scenario reports accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. | Stanford CRFM | Composite indices | University | Not in cards | | Nov 2022 | Jul 2023 | ||
| Hermes Bench Agents in Hermes Agent do 150 tasks across 25 categories, such as research, diagrams, memory, tool use and safety. Graders check the files and state left behind. | Nous Research | General assistants | Expert | Not in cards | Claude Opus 5.5 76.7% | Oct 2026 | Nov 2026 | ||
| Hermes Index Nous Research averages pass@1 and cost per task over Hermes Bench, Terminal-Bench 4.0, Terminal-Bench Science and SkillsBench, all run in Hermes Agent. | Nous Research | Composite indices | Expert | Not in cards | Claude Opus 5.5 63.3 | Oct 2026 | Nov 2026 | ||
| IndustryBench Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses. | Alibaba (1688 multimodal and industrial AI) | Other professions | Expert | Not in cards | | May 2026 | Mar 2026 | ||
| Insurance Agent Benchmark Models read commercial insurance documents and answer questions about their contents. Cooper compares each model alone and with its document processing system. | Cooper Labs | Documents, charts and text in images | University | Not in cards | Grok 4.6 77.6% | Sep 2026 | Sep 2026 | ||
| KorNAT Models answer questions about Korean social values and common knowledge. The authors compare answers with Korean reference judgments. | Jiyoung Lee and colleagues | Knowledge in many languages | Everyday | Not in cards | Claude Opus 5 65.4% | Feb 2024 | Aug 2026 | ||
| LatchBio Capabilities V0 Agents complete biological data analysis tasks. The historical score averages the available benchmark scores equally and excludes missing scores. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Sonnet 5.5 51.3% | Oct 2026 | |||
| Legora BAR Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers. | Legora | Law | Expert | Not in cards | Claude Opus 5.5 106.0 | Sep 2026 | Oct 2026 | ||
| LLM Deceptiveness and Gullibility Models try to deceive other models and recognize deceptive claims. | Lech Mazur | Multi-agent games | Everyday | Not in cards | Claude 3.5 Sonnet 1.10 | Mar 2025 | |||
| LLM Physics Plotting Benchmark Models write Python code to produce requested plots from ATLAS collision data. Each trial passes when the code runs and meets the configured checks. | Gordon Watts | Lab and data work | University | Not in cards | GPT-5.2 Codex 90.6% | Sep 2025 | Jul 2026 | ||
| MaxxBench Agents solve 89 modified DeepSWE repository tasks. Revised requirements, code, reference patches, and tests check whether agents can solve changed problems rather than recall original fixes. | unLAP | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 76.4% | Oct 2026 | Oct 2026 | ||
| Multi-Crit Multimodal judges compare responses under several criteria. Human judgments provide the reference for each criterion and reveal trade-offs between criteria. | Multi-Crit authors | General visual questions | University | Not in cards | o4-mini 43% | Nov 2025 | Oct 2025 | ||
| Multimodal RewardBench 2 Reward models compare two outputs containing text and images. Accuracy measures agreement with expert preferences across four tasks. | Meta and Surge AI | General visual questions | University | Not in cards | | Dec 2025 | Nov 2025 | ||
| Multimodal RewardBench Original Replaced in Dec 2025 | Meta and collaborators | Not in cards | | Feb 2025 | Sep 2024 | ||||
| NarrativeBench Models seek, include, exclude, and recover narratives from multilingual news articles. Citation-based F1 or recall scores narrative coverage after checking the response language. | Nikhil Sharma, Kelly Marchisio, Kenton Murray, and Ziang Xiao | Reading in many languages | Everyday | Not in cards | | Nov 2026 | Nov 2026 | ||
| OGS-Bench Models use a supplied culinary vocabulary to choose food and drink pairings, classify dishes, and produce structured answers. The headline score combines automatically checked tasks. | Gastrona | Language reasoning | University | Not in cards | GPT-5.5 86.8 | Apr 2026 | Apr 2026 | ||
| OOLONG Models aggregate information across long collections of text. The hub port implements the synthetic and real datasets with a context-controlled recursive language model interface. | OOLONG authors | Reasoning over long documents | University | Not in cards | | Sep 2025 | Apr 2026 | ||
| Open Persian LLM Leaderboard Models answer Persian multiple-choice questions. The leaderboard averages accuracy across five tasks, including two question sets from the organizers. | Part AI and Amirkabir University of Technology NLP Lab | Knowledge in many languages | University | Not in cards | | Nov 2024 | Mar 2025 | ||
| Passport Control Models maintain a dated memory across chats, emails, and tool results. Tests check whether they retain, update, expire, and retrieve facts at the right time. | Rairo Mukamuri | Finding facts in long inputs | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| PGG-Bench Agents decide how much to contribute to a public good and whether to punish other players. | Lech Mazur | Multi-agent games | Everyday | Not in cards | | Mar 2025 | |||
| Realm Financial Reasoning Agents analyze financial documents and spreadsheets. A model grades their answers against practitioner rubrics. | micro1 | Finance | Expert | Not in cards | | Sep 2026 | |||
| ReSpaceBench Models recognize rooms across separate visits and track objects that move between visits. The authors test 32 models on synthetic and real homes. | Tsinghua University and ByteDance | Embodied reasoning | Everyday | Not in cards | GPT-5.6 Sol 50.5% | Nov 2026 | Aug 2026 | ||
| SAFE-Teen Chatbots hold 6 to 20 turn chats with a simulated teen across 106 clinician-written scenarios, branching on how the teen reacts. A judge rates safety checks and reports the failure rate. | Vals AI | Wellbeing and sycophancy | Everyday | Not in cards | | Oct 2026 | Oct 2026 | ||
| SAFE-Teen Health advice Models respond to simulated teens in health advice scenarios. The score is the published conversation failure rate. | Vals AI | Wellbeing and sycophancy | Everyday | Not in cards | GPT-6.1 Sol 9.00% | Oct 2026 | Oct 2026 | ||
| SAFE-Teen Relationship boundaries Models respond to simulated teens in relationship boundary scenarios. The score is the published conversation failure rate. | Vals AI | Wellbeing and sycophancy | Everyday | Not in cards | GPT-6.1 Sol 19% | Oct 2026 | Oct 2026 | ||
| SAFE-Teen Urgent safety Models respond to simulated teens in urgent safety scenarios. The score is the published conversation failure rate. | Vals AI | Wellbeing and sycophancy | Everyday | Not in cards | GPT-6.1 Sol 15% | Oct 2026 | Oct 2026 | ||
| ScienceBoard Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking. | OS-Copilot team (Shanghai AI Laboratory and HKU) | Desktop apps | Expert | Not in cards | Claude Opus 4.8 68% | May 2025 | Aug 2026 | ||
| Slop Score Model writing is scored by how often it uses words and patterns that are far more common in AI text than in human text. Lower means less slop. | EQ-Bench (Sam Paech) | Taste and style | Everyday | Not in cards | Kimi K2 18.3 | Oct 2025 | Sep 2025 | ||
| SPM-Bench Models interpret microscopy images and select all correct answers. The primary score gives partial credit and penalizes incorrect selections. | Alibaba Group and Skylenage | Lab and data work | Expert | Not in cards | Claude Opus 4.7 0.94 | Feb 2026 | Apr 2026 | ||
| Stick to Your Role! Models simulate populations of fictional personas and answer personal-values questionnaires in many different conversation contexts. The benchmark measures how stable the expressed values stay across contexts, both for each persona and for the ranking of personas within the population. | Inria (Flowers team) and collaborators | Creative writing | Everyday | Not in cards | | Feb 2024 | Apr 2025 | ||
| The Aggregate Personal Style Imitation Models rewrite 53 neutral drafts in one person's style using 30 other messages as examples. An authorship classifier scores each rewrite. | Mikhail Doroshenko (The Aggregate) | Creative writing | Everyday | Not in cards | Claude Fable 5.1 76.7% | Nov 2026 | Nov 2026 | ||
| ThinkGeo Agents use remote sensing tools to interpret aerial and satellite images. The authors compare individual tool calls and use GPT-4o mini to grade final answers. | Mohamed bin Zayed University of Artificial Intelligence and collaborators | Apps and MCP servers | University | Not in cards | GPT-6 Sol 100% | May 2025 | Oct 2026 | ||
| UltraText Bench Image generators render dense bilingual text in many regions from 432 prompts. Scores check string accuracy, placement and legibility. | Westlake University | General visual questions | Everyday | Not in cards | GPT-Image-2 99.3 | Oct 2026 | Feb 2026 | ||
| VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. | Scale AI | Function calling | University | Not in cards | | Oct 2025 | Jul 2026 | ||
| 4DCodeBench Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion. | 4DCodeBench team | Physical prediction | Expert | Not in cards | GPT-6 Astra 0.79 | Oct 2026 | Sep 2026 | ||
| Accelerate Bench Coding agents speed up 32 real scientific applications, such as GROMACS and OpenFOAM, on a fixed CPU allocation within 8 hours. A hidden verifier checks correctness, and Score@1 averages the share of runtime saved, with incorrect programs scoring zero. | Tencent Hy Team, HKUST, HKUST (Guangzhou), Tsinghua University and Peking University | Speed and optimization | Expert | Not in cards | Claude Sonnet 5.5 67.4% | Nov 2026 | Nov 2026 | ||
| Active-SWE Coding agents inspect repository snapshots and fix bugs without issue reports. Evaluation checks patches against reference tests and measures localization precision and recall. A separate stage uses generated tests and model review to assess newly revealed bugs. | Sichuan University and University of Electronic Science and Technology of China | Fixing real repositories | Expert | Not in cards | Claude Opus 4.8 20% | Aug 2026 | Jul 2026 | ||
| ANIMA Models explain decisions about animal welfare. A judge scores their reasoning across 13 ethical dimensions. | Compassion Aligned Machine Learning and Sentient Futures | Deception and alignment | Everyday | Not in cards | | Mar 2026 | Jun 2026 | ||
| BalatroBench Models play the deck-building roguelike Balatro through an API bot. The leaderboard compares how far each model and strategy gets in a run. | Coder | Video games | Everyday | Not in cards | | Aug 2025 | Jan 2026 | ||
| CiteVQA Multimodal models answer questions about long PDFs and must cite the page region that supports each answer with a bounding box. The set has 1,897 questions over 711 documents in seven domains and two languages, averaging 40.6 pages per document. Strict Attributed Accuracy counts an answer only when both the answer and the cited region are correct. | OpenDataLab (Shanghai AI Laboratory) | Documents, charts and text in images | University | Not in cards | | May 2026 | Apr 2026 | ||
| Clash of AIs Season 1 Models publish trade entries, targets, and stops before prices move. Clash of AIs simulates fills against market prices and ranks accounts by total equity. | Clash of AIs | Finance | University | Not in cards | GPT-5.6 Sol $283,903 | Apr 2026 | Sep 2026 | ||
| Claw-SWE-Bench General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches. | TokenRhythm | Fixing real repositories | Expert | Not in cards | | Jun 2026 | Sep 2026 | ||
| CliniCARE-Bench Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases. | Scale AI | Medicine | Expert | Not in cards | GPT-5.5 76.1% | Aug 2026 | Jul 2026 | ||
| Crest Arcade Nations Royale Models manage one nation against 47 computer opponents on a simulated map. Wins earn full credit and unfinished games earn credit for territory held. | Glimmer Index | Video games | Everyday | Not in cards | gpt-oss-20b 100.0 | Nov 2026 | |||
| CRM-Agent-Bench Agents handle customer and business requests in a simulated enterprise. Executable checks verify tool arguments, action order, and which records changed across 555 tasks. | Salesforce | Running a business | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| DAYJOB: Data Science Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Data science and SQL | Expert | Not in cards | Claude Opus 5.5 37.3% | Oct 2026 | Oct 2026 | ||
| DAYJOB: Tax Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria. | Surge AI | Finance | Expert | Not in cards | Claude Sonnet 5.5 10.8% | Oct 2026 | Oct 2026 | ||
| Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. | Snorkel AI | Composite work indices | Expert | Not in cards | | Oct 2026 | Oct 2026 | ||
| EmoBench-M Multimodal models are tested on emotional intelligence across 13 scenarios in three dimensions: emotion recognition, conversational emotion understanding and socially complex emotion analysis. | EmoBench-M authors | General visual questions | Everyday | Not in cards | | Feb 2025 | Dec 2025 | ||
| FeatEng Models write Python functions to transform 103 tabular datasets. The score measures how much those changes reduce prediction error for an XGBoost model. | Snowflake and Institute of Computer Science, Polish Academy of Sciences | Data science and SQL | Expert | Not in cards | o1-preview 11.1% | Oct 2024 | Sep 2024 | ||
| FinDesk-Bench Agents find Treasury figures and use them to handle client requests across ten business services. A model grades API calls and the records agents leave behind. | Veris AI | Documents and spreadsheets | University | Not in cards | | Nov 2026 | Nov 2026 | ||
| GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. | SemiAnalysis | Tasks from real jobs | Expert | Not in cards | Claude Fable 5.2 16.4% | Nov 2026 | Nov 2026 | ||
| GIM Models solve problems that combine constraint tracking, reasoning, and communication using broadly accessible knowledge. Model judges score answers against task rubrics. An item response model adjusts for problem difficulty and judge effects to estimate ability. | Rohit Patel, Alexandre Rezende, and Steven McClain at Meta | Language reasoning | Everyday | Not in cards | GPT-5.4 Pro 1.63 | May 2026 | Apr 2026 | ||
| HearingBench Models identify sounds from spectrogram images without hearing the audio. The test covers everyday sounds, instruments, vocals, words and sentences. | Adam Holter | Audio understanding and voice | University | Not in cards | | Oct 2026 | Nov 2026 | ||
| HireWatch Models select questions for job interviews under different workplace pressures. The benchmark measures how often they select questions classified as unlawful under US employment law. | Huy Nghiem and colleagues | Law | University | Not in cards | Claude Opus 4.7 0.05% | Nov 2026 | Apr 2026 | ||
| IndicIFEval Models follow instructions in 14 Indic languages. Rule-based checks test constraints such as required words, paragraph structure and word counts. | AI4Bharat | Chat and instructions in many languages | Everyday | Not in cards | GPT-5 82.4% | Feb 2026 | Dec 2025 | ||
| Inductive Arithmetic Pattern Reasoning Models infer a hidden arithmetic operation from worked examples and calculate the result for a new expression. The grader checks the final integer. | Indra Agustian | Abstract puzzles and logic | School | Not in cards | | Nov 2026 | Oct 2026 | ||
| InSAR Regional Snow Mapping (Axis Spatial) Agents analyze Alpine radar scenes, align weather observations with radar windows, and calculate regional statistics. A script supplies reference answers for 24 tasks. | Axis Spatial | Lab and data work | University | Not in cards | Claude Opus 4.7 100% | Nov 2026 | Oct 2026 | ||
| LEGO-Puzzles Vision-language models answer LEGO assembly questions, from basic spatial understanding to planning several assembly steps. The elementary set has 1,100 questions over 11 tasks. | Shanghai AI Laboratory and collaborators (Tang, Duan et al.) | Spatial reasoning and counting | Everyday | Not in cards | GPT-5 72% | Mar 2025 | Oct 2025 | ||
| MCP-Bench Agents solve multi-step tasks using 28 live MCP servers with 250 tools in areas such as finance, travel, scientific computing and academic search. Scoring covers tool use, coordination and planning. | Accenture | Apps and MCP servers | Expert | Not in cards | GPT-5 0.75 | Aug 2025 | Aug 2025 | ||
| MediaPerf Models tag and summarize video advertisements. The benchmark measures tag accuracy, summary quality, cost and throughput. | Coactive AI | Video understanding | University | Not in cards | GPT-5.4 0.42 | Mar 2026 | Apr 2026 | ||
| MMLU Medical Models answer the multiple-choice questions from the medical subjects of MMLU, such as anatomy, clinical knowledge and professional medicine. | UC Berkeley (Hendrycks et al.), subset chosen by medical model reports | Medicine | University | Not in cards | | Sep 2020 | May 2024 | ||
| Multi-LCB Models solve LiveCodeBench-style contest problems in twelve programming languages, with dated problems for contamination-aware evaluation. | Multi-LCB team | Competitive programming | Competition | Not in cards | gpt-oss-120b 67.8% | Jun 2026 | Aug 2025 | ||
| OWE-Bench 26 long-horizon engineering and research projects across software, physical engineering, machine learning and scientific computing, from GPU kernels to aircraft design. Agents get 4 to 18 hours per attempt, and an offline verifier scores what they ship on held-out cases. | Mercor | Building whole projects | Research frontier | Not in cards | GPT-6.2 Astra 67.2% | Nov 2026 | Nov 2026 | ||
| PianoBench Models answer questions about piano performances from recordings or symbolic transcriptions. The authors compare accuracy with and without access to the music. | Carnegie Mellon University, Queen Mary University of London, and Cornell University | Audio understanding and voice | University | Not in cards | GPT-6 Astra 93.2% | Nov 2026 | Sep 2026 | ||
| RadLE 2.0 Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors. | CRASH Lab, Koita Centre for Digital Health, Ashoka University | Medicine | Expert | Not in cards | Claude Fable 5 758.0 | Jul 2026 | Jul 2026 | ||
| Radiology's Last Exam (RadLE) 1.0 Replaced in Jul 2026 | Not in cards | GPT-5 30% | Sep 2025 | Aug 2025 | |||||
| SG-LegalClarify Models answer Singapore legal questions and ask for missing facts. Three model judges grade each conversation against rubrics written by lawyers. | Ziwei Gu and colleagues | Law | Expert | Not in cards | Kimi K3 77.8% | Nov 2026 | Jul 2026 | ||
| SMB Bench An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule. | ORO AI | Running a business | University | Not in cards | GPT-6 Astra $7,749 | Nov 2026 | Nov 2026 | ||
| SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. | Snorkel AI | Law | Expert | Not in cards | | Oct 2026 | Oct 2026 | ||
| SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. | Snorkel AI | Other professions | Expert | Not in cards | | Oct 2026 | Oct 2026 | ||
| SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. | Snorkel AI | Running a business | Expert | Not in cards | | Oct 2026 | Oct 2026 | ||
| SpaceChem Bench Agents program reactors to assemble molecules in SpaceChem. The main ranking measures how close their solutions come to the community’s records. | Wincon Research | Puzzles and chess | Competition | Not in cards | GPT-6 Astra 74.3 | Nov 2026 | Oct 2026 | ||
| SuperCLUE-Project Models generate code for complete software projects. | SuperCLUE | Building whole projects | Expert | Not in cards | o3-mini 82.1 | Apr 2025 | |||
| SuperCLUE-Science Models solve Chinese science reasoning problems. | SuperCLUE | Science questions | University | Not in cards | o4-mini 80.8 | Apr 2025 | |||
| SuperCLUE-SWE Agents complete software engineering tasks in Chinese. | SuperCLUE | Fixing real repositories | Expert | Not in cards | Claude Opus 4.8 73.3 | Jun 2026 | |||
| SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result. | Snorkel AI | Fixing real repositories | Research frontier | Not in cards | Claude Opus 5.5 16.6% | Oct 2026 | Oct 2026 | ||
| SWE-CI Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes. | SKYLENAGE AI | Fixing real repositories | Expert | Not in cards | Claude Opus 4.6 92% | Mar 2026 | Feb 2026 | ||
| SynthBench Models answer survey questions. The index compares their answer distributions with human responses across four datasets, using random answers and human agreement as reference points. | DataViking | Social simulation | Everyday | Not in cards | | Apr 2026 | Oct 2026 | ||
| TeachArena Agents make teaching decisions, tutor students through dialogue, and carry out learning management system tasks. The evaluator checks conversation turns, artifacts, and final system states. | The Hong Kong University of Science and Technology and Alibaba | Other professions | University | Not in cards | Claude Opus 4.8 0.80 | May 2026 | Jun 2026 | ||
| TIS-100 Bench Agents write and optimize assembly programs for 55 puzzles. The ranking counts puzzles where a program improves on the community’s recorded tradeoffs in cycles, nodes, and instructions. | Wincon | Speed and optimization | Competition | Not in cards | GPT-6 Astra 11.0 | Nov 2026 | Oct 2026 | ||
| UniCode Models solve generated competitive programming problems using automated test cases. Harbor packages 399 problems with validated reference solutions. | Xinyue Zheng and colleagues | Competitive programming | Competition | Not in cards | o4-mini 70.3% | Oct 2025 | Sep 2025 | ||
| ViMU Models explain what a short video implies and select the cues that support their reading. They also identify rhetorical devices and social signals. | National University of Singapore | Video understanding | Everyday | Not in cards | o4-mini 46.9% | May 2026 | Mar 2026 | ||
| VisionArena-Bench Models answer visual questions from Arena users. A vision language model judges their answers to approximate human preference rankings. | Arena | General visual questions | Everyday | Not in cards | | Dec 2024 | Oct 2024 | ||
| AgroMind (original) Models answer questions about farmland images across 13 tasks. They identify crops, assess crop health and interpret spatial relationships. | Qingmei Li and colleagues | General visual questions | University | Not in cards | | May 2025 | Dec 2024 | ||
| AidanBench Models keep giving new answers to open-ended questions until answers become incoherent or repeat earlier ones. The count of fresh, coherent answers is the score, with no ceiling. | Aidan McLaughlin (community) | Creative writing | Everyday | Not in cards | o1 3204.0 | Aug 2024 | Feb 2025 | ||
| Biopharma Bench Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers. | Raycaster | Genomics and life sciences | Expert | Not in cards | GPT-6 Astra 65.1% | Sep 2026 | Nov 2026 | ||
| CLASHarness Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment. | Google Research | Desktop apps | Everyday | Not in cards | GPT-6 Astra 28.7% | Oct 2026 | Sep 2026 | ||
| Code2Video-Compass Multimodal models write programs that render complete videos from text, an image or a video. The default Overall board ranks models by their mean rank across quality metrics, so a lower score is better. | Fudan University | Video generation | Everyday | Not in cards | Claude Opus 5.5 1.07 | Oct 2026 | Oct 2026 | ||
| Compass Academic Leaderboard Models answer academic reasoning tasks. OpenCompass publishes model rankings and predictions across the underlying evaluations. | OpenCompass | Composite indices | University | Not in cards | | Dec 2024 | Mar 2025 | ||
| DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. | DeepPHY authors | Abstract puzzles and logic | Everyday | Not in cards | Claude 3.7 Sonnet 39% | Aug 2025 | Jun 2025 | ||
| FailSafeQA Models answer finance questions over long documents while the query or the document is degraded, irrelevant or missing. An LLM judge scores robustness, context grounding and compliance. | Writer | Finance | Expert | Not in cards | | Feb 2025 | Mar 2025 | ||
| FinFIRST Agents research financial questions and provide supporting sources. The benchmark grades individual requirements and reports how often an answer meets every requirement. | Wenqing Wang and colleagues | Finance | Expert | Not in cards | GPT-5.6 Sol 71.5% | Sep 2026 | Sep 2026 | ||
| GameXpert-Bench Agents build games from 97 briefs, repair bugs in 100 tasks, and improve 17 games over six requests each. Each track reports its own scores. | Tencent Hunyuan and Lightspeed Studios | Building whole projects | University | Not in cards | Claude Opus 5 70.9 | Aug 2026 | Jul 2026 | ||
| GeoGuessBench (panoramic) Models see four Google Street View images taken at right angles. They predict coordinates and report their confidence. Closer guesses earn more points. | GeoGuessBench (panoramic) | Spatial reasoning and counting | Everyday | Not in cards | | Feb 2026 | |||
| GSM1k Models solve new grade school math word problems written to mirror GSM8K in style and difficulty. The gap between GSM8K and GSM1k accuracy shows overfitting or contamination. | Scale AI | Word problems | School | Not in cards | Claude 3.5 Sonnet 96.6% | May 2024 | Aug 2024 | ||
| HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. | Microsoft Research | Medicine | Expert | Not in cards | Claude Opus 5.5 57.3% | Oct 2026 | Sep 2026 | ||
| HealthAgentBench v1 Replaced in Oct 2026 | Not in cards | Claude Opus 5 55% | Jun 2026 | Jul 2026 | |||||
| LEXam-hard Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed. | Joel Niklaus (LEXam co-author) | Law | Research frontier | Not in cards | | Sep 2026 | Jul 2026 | ||
| LOKI Models distinguish synthetic content from real content across images, videos, audio, text, and 3D data. The benchmark checks real-or-synthetic labels and annotated abnormal details, with separate tasks for explaining anomalies. | OpenDataLab | General visual questions | Everyday | Not in cards | LLaVA-OneVision 72B 60.8% | Oct 2024 | Aug 2024 | ||
| M-IFEval Models follow instructions in French, Japanese, and Spanish. Automatic checks test whether each answer satisfies the requested constraints. | Lightblue | Chat and instructions in many languages | Everyday | Not in cards | o1-preview 86.6% | Jul 2024 | Sep 2024 | ||
| MemGUI-Bench Agents remember details while working across Android apps. Tasks include recording audio and creating calendar events. A model checks the actions and results across repeated attempts. | Zhejiang University, Nankai University, Chinese University of Hong Kong, Shanghai Jiao Tong University, and vivo AI Lab | Phones | Everyday | Not in cards | | Feb 2026 | Aug 2026 | ||
| MVDriveBench Models answer driving questions using synchronized videos from cameras facing different directions. The authors compare surround-view results with front-camera results. | Shunming Liu and colleagues | Embodied reasoning | Everyday | Not in cards | GPT-5.5 80.1% | Nov 2026 | Apr 2026 | ||
| OBI-Bench Models locate inscriptions, rejoin bone fragments, classify characters, retrieve matches, and explain glyphs. The authors publish separate scores for each task. | Shanghai Jiao Tong University | Documents, charts and text in images | Expert | Not in cards | GPT-4o 0.61 | Oct 2024 | Aug 2024 | ||
| OccuBench Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable. | Qwen (Alibaba) | Tasks from real jobs | Expert | Not in cards | GPT-5.2 79.6% | Apr 2026 | Mar 2026 | ||
| OmicsBench v1 Agents analyze molecular data across eight benchmarks. LatchBio reports pass rates for 741 short tasks and 43 long tasks. | LatchBio | Genomics and life sciences | Expert | Not in cards | GPT-6 Astra 40.8% | Oct 2026 | Sep 2026 | ||
| OPT-BENCH Agents iteratively improve solutions to 20 machine-learning tasks and 10 combinatorial optimization problems using execution feedback. Win counts, invalid-solution rates, ranks, and improvement rates measure optimization performance. | Tongji University, Shanghai AI Laboratory, Nanjing University and others | Scientific code | University | Not in cards | | Jun 2025 | Aug 2025 | ||
| OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. | Shanghai AI Laboratory (InternLM) | Video understanding | Everyday | Not in cards | | Jun 2026 | Apr 2026 | ||
| ParlaCAP Test Datasets Models read parliamentary speeches and choose a policy topic or Other. The four-language test compares predictions with human labels on 3,443 speeches. | ParlaCAP project | Reading in many languages | School | Not in cards | GPT-5 0.69 | Jun 2025 | Sep 2025 | ||
| PISA-Bench Models answer illustrated PISA questions in six languages. Parallel questions let researchers compare language effects on reading, science and mathematical reasoning. | Patrick Haller and colleagues | General visual questions | School | Not in cards | | Oct 2025 | Oct 2025 | ||
| PoETa v2 Models answer multiple-choice and open-ended questions in Portuguese. The hub port selects native Portuguese tasks covering inference, reading, exams, proverbs and toxicity. | PoETa v2 team | Knowledge in many languages | University | Not in cards | GPT-4.1 76.2 | Sep 2025 | Apr 2025 | ||
| PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. | Novee | Finding and exploiting real bugs | Expert | Not in cards | Grok 4.6 63.3% | Aug 2026 | Oct 2026 | ||
| Retrobench v0.4 Agents reconstruct C from five Nintendo 64 assembly targets. The organizer scores the average percentage of matching bytes under a budget of 25 compilations per target. | Sinan Onur Altınuç | Function-level problems | Expert | Not in cards | | Aug 2026 | Nov 2026 | ||
| RSI-Inference Bench Coding agents add model support to inference engines, improve serving speed and fix performance regressions. The harness builds and runs each submission. | Apple | Machine learning engineering | Expert | Not in cards | Claude Sonnet 5.5 2,044 | Oct 2026 | Oct 2026 | ||
| SATBench Models decide whether a set of constraints in a narrative puzzle can all hold at once. A Boolean satisfiability solver supplies the correct label. | Anjiang Wei and colleagues | Language reasoning | University | Not in cards | o4-mini 89.3% | May 2025 | Apr 2025 | ||
| Sinhala Language Proficiency & Linguistic Mastery Models answer Sinhala prompts about grammar, style, conversation, and literature. Three judges score the responses without seeing the model names. | Nadew Damruwan | Chat and instructions in many languages | School | Not in cards | GPT-6 Astra 95.9 | Nov 2026 | Sep 2026 | ||
| TCSExam-Lite Agents solve 20 theoretical computer science tasks, five per track. The default solver protocol gives hidden feedback only after submission. | Moonshot AI | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 72.3 | Nov 2026 | Oct 2026 | ||
| TCSExam Full Replaced in Nov 2026 | Not in cards | GPT-6 Astra 71.9 | Nov 2026 | Oct 2026 | |||||
| The Aggregate Exotic Fact Recall Models recall exact fields from five public registries without looking them up. The bank mixes familiar entries with randomly sampled obscure ones. | Mikhail Doroshenko (The Aggregate) | Trivia and open questions | Expert | Not in cards | Claude Fable 5.2 75% | Sep 2026 | Nov 2026 | ||
| Thrive Holdings helpdesk ticket resolution benchmark Agent investigates IT support requests, takes action and decides when to escalate to a technician. Scored by pass rate. | Thrive Holdings | Tasks from real jobs | University | Not in cards | GPT-6 Astra 61.5% | Oct 2026 | Sep 2026 | ||
| Video-SafetyBench Video-language models receive 2,264 synthesized videos paired with a harmful or a benign-looking query across 13 unsafe categories and 48 subcategories. An LLM judge scores harm with a confidence-weighted metric called RJScore. | BAAI FlagEval team | Jailbreaks and prompt injection | Everyday | Not in cards | Claude 3.7 Sonnet 1.30 | May 2025 | Mar 2025 | ||
| AML Bench (Veris AI) Agents investigate suspicious activity at a simulated bank. A model judge checks whether the recorded service requests and final case disposition satisfy every requirement. | Veris AI | Finance | Expert | Not in cards | GPT-6.1 Sol 70.8% | Oct 2026 | Nov 2026 | ||
| AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. | Allen Institute for AI (Ai2) | Research science | Expert | Not in cards | Claude Opus 4.7 58% | Aug 2025 | Apr 2026 | ||
| BhashaBench-Multi (Krishi) Models answer agricultural exam questions in 22 Indian languages. The evaluation checks the selected answer against a key for each translated question. | BharatGen | Knowledge in many languages | University | Not in cards | Jun 2026 | ||||
| BhashaBench-Krishi V1 Replaced in Jun 2026 | Exam questions | Not in cards | | Oct 2025 | Aug 2025 | ||||
| cad-spec Models write CadQuery code for mounting plates from dimensioned specifications. The evaluator measures each finished solid against nine requirements. | cad-spec team | CAD models | University | Not in cards | | Aug 2026 | Feb 2026 | ||
| CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. | OpenCompass (Shanghai AI Laboratory) | Composite indices | Expert | Not in cards | Claude Opus 5 83.3 | Sep 2023 | Aug 2026 | ||
| Crest Arcade Nonograms Models reconstruct a picture from row and column clues in one reply. The score averages the share of exact solutions across six tiers. | Glimmer Index | Puzzles and chess | Everyday | Not in cards | | Nov 2026 | |||
| Crest Arcade Ravelin Models build towers to protect a keep through waves of attackers. The score compares wins with the reference time plus a penalty for lives lost. | Glimmer Index | Video games | Everyday | Not in cards | | Nov 2026 | |||
| Crest Arcade Starjar Models place pieces in a jar so matching pieces touch and merge. The score compares their total points with a reference bot on identical jars. | Glimmer Index | Video games | Everyday | Not in cards | | Nov 2026 | |||
| Crest Arcade Wildhaven Models select a team of animals and choose where each lands in a village. The score measures the share of village strength they destroy. | Glimmer Index | Video games | Everyday | Not in cards | | Nov 2026 | |||
| Cresta-Bench Contact-center agents take simulated calls for lending, health and airline lines. Tool calls get deterministic checks and LLM judges grade the rest. | Cresta | Customer service | Everyday | Not in cards | Claude Opus 5.5 82.6% | Oct 2026 | Oct 2026 | ||
| FrontierBioML 25 machine learning tasks in computational biology, across antibody development, protein engineering, structural biology, binder design and functional genomics. An agent gets real experimental data and a brief, works offline for up to six hours and submits a prediction script. A sealed grader runs the script on held-out data, such as new antibody families or new protein targets, and scores each task from 0 at a simple baseline to 100 for a perfect answer. | Valen Labs | Genomics and life sciences | Expert | Not in cards | Claude Mythos 5.2 40.3 | Nov 2026 | Nov 2026 | ||
| GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Scored by a distance-based GeoGuessr-style reward from 0 to 1. | FineEnvs (Hugging Face community) | General visual questions | Everyday | Not in cards | Claude Sonnet 5 0.64 | Sep 2026 | Jun 2026 | ||
| GSM-Infinite Models solve generated arithmetic problems across different context lengths and numbers of reasoning steps. The benchmark varies how many irrelevant details appear alongside each problem. | Carnegie Mellon University and Meta AI | Reasoning over long documents | School | Not in cards | | Feb 2025 | May 2025 | ||
| HealthQA-BR Models answer Portuguese multiple-choice questions about clinical care and public health in Brazil’s Unified Health System. | Andrew Maranhão Ventura D'addario | Medicine | Expert | Not in cards | GPT-4.1 86.6% | Jun 2025 | Apr 2025 | ||
| HoloBench Models answer questions by combining facts across database rows written as sentences. A judge gives full, partial, or zero credit for each expected result. | Megagon Labs | Reasoning over long documents | School | Not in cards | | Oct 2024 | Feb 2025 | ||
| Incident Arena Agents diagnose and repair incidents in running services. The score averages the repair success rate across 20 incidents. | Abundant AI and collaborators | Terminal tasks | Expert | Not in cards | GPT-6 Astra 64.3% | Sep 2026 | Oct 2026 | ||
| InsureBench Multi-functional LLMs Models read insurance files and answer questions about claims, coverage, and actuarial decisions. InsureThing combines automatic answer checks with rubric grades for judgment tasks. | InsureThing | Finance | University | Not in cards | Claude Opus 4.8 96.3% | Jun 2026 | Jun 2026 | ||
| ITSMBench Agents operate an IT service desk across 53 conversations with simulated colleagues. The verifier grades the resulting database state, including permissions and required actions. | Vibrant Labs | Customer service | University | Not in cards | Grok 4.6 0.70 | Aug 2026 | Aug 2026 | ||
| LegalCiteBench Models recover and verify citations from United States judicial opinions without external lookup. The benchmark also tests whether models match cases and correct citation errors. | Sijia Chen and colleagues | Law | University | Not in cards | Claude Sonnet 4.5 0.78 | Feb 2026 | Oct 2025 | ||
| Letta Leaderboard Agents read, write and update core and archival memory while answering synthetic factual questions. A model judge scores answers and penalizes unnecessary memory operations. | Letta | Learning from the context | Expert | Not in cards | GPT-5.2 Codex 93.0 | May 2025 | Mar 2026 | ||
| LingOly-TOO Models solve Linguistics Olympiad problems whose spelling patterns have been systematically changed while preserving the solution logic. Answers receive exact-match credit with no partial credit. The evaluation compares performance on original and obfuscated problems. | University of Oxford and collaborating linguistics researchers | Reasoning in many languages | Competition | Not in cards | GPT-5 46.7% | Mar 2025 | Sep 2025 | ||
| LLM Uncertainty Bench Models answer questions across five language tasks. The original benchmark measures accuracy and uncertainty, while the port checks only the selected answer. | Tencent AI Lab, University College London, and University of Macau | Language reasoning | University | Not in cards | | Jan 2024 | Nov 2023 | ||
| LongProc Models carry out long procedures such as converting HTML tables, tracing paths and tracking beliefs. The evaluator checks outputs as the required procedure grows. | Xi Ye and colleagues | Reasoning over long documents | University | Not in cards | | Jan 2025 | Oct 2024 | ||
| MM-FinEval Models forecast financial outcomes from earnings call transcripts, slides, and audio. The authors compare 19 models using directional accuracy and absolute error. | Northwestern University and collaborators | Finance | University | Not in cards | GPT-5.5 88% | Sep 2026 | May 2026 | ||
| MUSE (CAD assemblies) Models turn design specifications into CAD assemblies. A program checks the geometry. A vision model grades function, manufacturability, and ease of assembly. | Xiaoyu Dong and colleagues | CAD models | University | Not in cards | GPT-5.5 52.4% | May 2026 | Apr 2026 | ||
| Next.js Agent Evals Coding agents with their models do real Next.js tasks. The board reports success rate with and without an AGENTS.md file, plus duration and cost. | Vercel | Web and app front ends | Expert | Not in cards | Claude Fable 5.1 100% | Oct 2025 | Oct 2026 | ||
| OfficePrism Agents resolve Chinese workplace tickets using one business system at a time. The evaluator checks outcomes across 420 tasks in seven business functions. | Alibaba Group and collaborators | Documents and spreadsheets | University | Not in cards | Kimi K3 76.4% | Nov 2026 | Jul 2026 | ||
| OTelBench Coding agents add OpenTelemetry tracing, metrics and logging to codebases in several languages. Tests check the instrumentation. | Quesma | Fixing real repositories | Expert | Not in cards | Claude Opus 4.5 29% | Jan 2026 | Dec 2025 | ||
| PharmaBench (micro1) Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff. | micro1 | Medicine | Expert | Not in cards | Kimi K3.1 Pro 17.8% | Oct 2026 | Nov 2026 | ||
| RECON Research agents find and verify several facts about identified people on the open web without mixing up people with similar names. | Sixtyfour | Deep research | Expert | Not in cards | Grok 4.20 Multi-agent 36.2% | Apr 2026 | Sep 2026 | ||
| RPC-Bench Models read computer science papers and answer reviewers’ questions. A model judge grades correctness, completeness, and concision against verified review and rebuttal pairs. | Yelin Chen and colleagues | Research science | Expert | Not in cards | GPT-5 37.5 | Apr 2026 | Dec 2025 | ||
| RSNA Knee LLM Label Extraction Models extract twelve findings from knee MRI reports in a batch. The grader compares the labels with reference findings using macro ROC AUC. | Kirderf | Medicine | University | Not in cards | Claude Opus 5.5 0.89 | Oct 2026 | Sep 2026 | ||
| ScrabbleBench Models choose a word from a Scrabble rack and board. A solver finds its best legal placement. The score averages the four seats in self-play. | Sunny Balasubramanian | Puzzles and chess | Everyday | Not in cards | GPT-5 152.8 | Aug 2025 | Sep 2025 | ||
| Shopping MMLU Models complete 57 shopping tasks involving product concepts, reasoning, user preferences, and several languages. Task-specific metrics check classification, ranking, extraction, translation, and generated text. Skill scores average the corresponding task metrics. | Amazon and academic collaborators | Running a business | Everyday | Not in cards | Claude 3 Sonnet 72.6 | Oct 2024 | Apr 2024 | ||
| SuperCLUE-CoT Models solve Chinese reasoning problems that require several steps. | SuperCLUE | Language reasoning | University | Not in cards | o1 77.7 | Jan 2025 | |||
| SuperCLUE-Math6o Models solve Chinese elementary school math olympiad problems. | SuperCLUE | Competition problems with answers | Competition | Not in cards | | Jan 2025 | |||
| SuperCLUE-VLM Vision language models interpret images and answer questions in Chinese. | SuperCLUE | General visual questions | Everyday | Not in cards | | Nov 2026 | |||
| SWE-Bench Pro Private Agents resolve held-out software issues from the private SWE-Bench Pro split. The leaderboard reports the percentage of resolved tasks. | Scale AI | Fixing real repositories | Expert | Not in cards | | Sep 2025 | Jul 2026 | ||
| Taste-Bench Models choose the better next step at 502 decision points in software engineering and research trajectories. A question scores only when both option orders receive the correct answer. The overall score equally weights engineering and research. | City University of Hong Kong and Microsoft | Machine learning engineering | Expert | Not in cards | GPT-5.6 Sol 59.7% | Sep 2026 | Jul 2026 | ||
| Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search. | Tencent | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 81.6% | Jul 2026 | Oct 2026 | ||
| ToRR Models answer questions over tables given in several formats, and the board checks how much accuracy changes when the table format or prompt changes. | IBM Research (on HELM) | Language reasoning | University | Not in cards | Claude 3.5 Sonnet (October 2024) 49.9 | Feb 2025 | Mar 2025 | ||
| Vals Smith: torvalds/linux Agents modify torvalds/linux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 74.1% | Aug 2026 | Aug 2026 | ||
| VANTAGE-BENCH Models answer questions, describe events, and locate objects or actions in infrastructure-camera footage. Task-specific metrics measure answer accuracy, caption quality, localization, and tracking. | NVIDIA | Video understanding | Everyday | Not in cards | GPT-6 Astra 75.3 | Sep 2026 | Sep 2026 | ||
| VoxParity Voice agents choose tool calls after listening to speech. Paired recordings keep the words fixed while changing an audible cue that affects the correct action. | Bhavik Mangla | Audio understanding and voice | Everyday | Not in cards | | Sep 2026 | Sep 2026 | ||
| WorkBuddyBench-Office Agents edit office files and reconcile records across a workspace. One task requires updating an eleven-slide presentation from a spreadsheet while preserving unrelated slides. | Tencent | Documents and spreadsheets | University | Not in cards | Kimi K3.1 Pro 86.4% | Jul 2026 | Oct 2026 | ||
| WorkplaceAgents Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics. | Snorkel AI | Tasks from real jobs | Expert | Not in cards | | Oct 2026 | Oct 2026 | ||
| AI AgriBench Models answer 416 agronomy questions from agricultural extension publications. Three judges compare each answer with an expert-reviewed reference across four metrics. | AI AgriBench Consortium, University of Illinois Urbana-Champaign | Tasks from real jobs | University | Not in cards | GPT-5 98.1 | Jan 2026 | Nov 2025 | ||
| ATM-Bench Agents answer questions about a person’s history using images, videos, and email records. The Harbor package contains 31 hard questions with schema-guided memory. | Jingbiao Mei and colleagues | Finding facts in long inputs | University | Not in cards | Claude Opus 4.5 86% | Mar 2026 | Jun 2026 | ||
| AutoResearchBench Agents search the scientific literature, either tracking down one target paper through multi-step probing or collecting every paper that meets given conditions. | Lei Xiong, Kun Luo, Zheng Liu et al. | Deep research | Expert | Not in cards | Claude Opus 4.6 9.39% | Apr 2026 | Mar 2026 | ||
| Beyond Survival Models play social deduction games with human-aligned strategies and interaction patterns. | Beyond Survival authors | Multi-agent games | Everyday | Not in cards | GPT-5 58.7% | Nov 2026 | Sep 2025 | ||
| Bill of Lading OCR Benchmark Models transcribe 25 synthetic bills of lading. A fixed Gemini model extracts 21 fields from each transcription for comparison with the known values. | okraPDF | Documents, charts and text in images | Everyday | Not in cards | | Aug 2026 | Aug 2026 | ||
| BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. | Alibaba Group (Qwen, Alibaba Token Hub) + Zhejiang University + CUHK | CAD models | Expert | Not in cards | GPT-6 Astra 79.9 | Oct 2026 | Sep 2026 | ||
| BlueBench Agents solve enterprise tasks across knowledge, coding, security and business workflows. The benchmark aggregates capability scores from multiple public task suites. | IBM Research | General assistants | Expert | Not in cards | GPT-4.1 64.6 | Sep 2025 | Apr 2025 | ||
| CadEval Models write OpenSCAD code for single mechanical parts from text descriptions. Rendered parts are checked against reference geometry, scored by pass rate. | Will Patrick | CAD models | University | Not in cards | o3 74% | Apr 2025 | Apr 2025 | ||
| ChipBench Models write Verilog modules, debug circuits and generate reference models. Simulations compare the outputs with reference implementations. | Zhongkai Yu and colleagues | Electronics | Expert | Not in cards | Claude Opus 4.5 30.5% | Jan 2026 | Dec 2025 | ||
| ClawBench V2 Browser agents do 130 new everyday tasks on live websites under six supported harnesses. Request interception and a more lenient LLM judge score each run. | TIGER-AI-Lab (University of Waterloo) and NAIL Group | Web browsing | Everyday | Not in cards | Claude Opus 4.7 44.6% | May 2026 | Apr 2026 | ||
| ClawBench V1 Replaced in May 2026 | Not in cards | Claude Opus 4.6 61.4% | Apr 2026 | Mar 2026 | |||||
| CorporateBench Agents answer questions over the email archives of four synthetic companies, up to 232,692 documents. Answers are checked against each company's ground-truth knowledge graph. | Epiq AI Labs | Reasoning over long documents | Expert | Not in cards | GPT-6.1 Sol 82.2% | Aug 2026 | Oct 2026 | ||
| Embodied Agent Interface Models interpret goals, decompose them into subgoals, sequence actions, and predict state transitions in BEHAVIOR and VirtualHome. The evaluator checks task completion and distinguishes hallucinations, invalid actions, and planning errors. | Stanford Vision and Learning Lab | Embodied reasoning | Everyday | Not in cards | o1-preview 70.3 | Oct 2024 | Sep 2024 | ||
| English to Icelandic Test Suite for Syntactic Divergence Models translate 345 English sentences into Icelandic. Human evaluators judge whether each translation uses an acceptable Icelandic construction for the targeted English syntax. | Árni Magnússon Institute for Icelandic Studies and University of Iceland | Translation | Everyday | Not in cards | GPT-5.5 93.4% | Jun 2026 | May 2026 | ||
| European LLM Leaderboard (OpenGPT-X) Models answer exam, science, commonsense, truthfulness, and math questions across European languages. The leaderboard also includes multilingual reading comprehension. | OpenGPT-X | Knowledge in many languages | School | Not in cards | | Oct 2024 | Jul 2024 | ||
| Factory Code Review Benchmark Models review real pull requests. Factory measures precision and recall against reference issues and reports mean F1. | Factory | Fixing real repositories | Expert | Not in cards | GPT-5.2 60.5% | Apr 2026 | Apr 2026 | ||
| Galileo LLM Hallucination Index RAG Special Models answer questions using short, medium, and long passages. Galileo judges whether each answer follows the supplied context. | Galileo | Faithful to sources | School | Not in cards | Claude 3 Opus 0.99 | Jul 2024 | Jun 2024 | ||
| Galileo LLM Hallucination Index v2023 Replaced in Jul 2024 | Hallucination rates | University | Not in cards | GPT-4 0.79 | Nov 2023 | Nov 2023 | |||
| GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. | Shanghai AI Laboratory (uni-medical) | Medicine | Expert | Not in cards | | Aug 2024 | Apr 2024 | ||
| Holdings-IndividualTaxBench 88 real client document packages turned into Form 1040 fields. Fact-level accuracy against practitioner-reviewed returns, with an LLM judge matching values. | Thrive Holdings | Tasks from real jobs | Expert | Not in cards | Claude Opus 5.5 76.3% | Oct 2026 | Sep 2026 | ||
| IITP-Legal Models translate 5,000 English legal sentences into Hindi. The suite includes 100 sentences at each length from five through 54 words. The original ranking uses BLEU. | AI and NLP Research Group, IIT Patna | Translation | University | Not in cards | | Jun 2025 | Jun 2025 | ||
| ITBench Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. | IBM Research | Terminal tasks | Expert | Not in cards | | Feb 2025 | Nov 2025 | ||
| LegacyShop Models maintain a Python repository for an online shop. Tasks test bug fixes, memory reasoning, complexity analysis, and whether to ask for clarification. | Iseoluwa Olowogoke | Fixing real repositories | University | Not in cards | Claude Opus 5.5 0.91 | Oct 2026 | Oct 2026 | ||
| ManyIH-Bench Models resolve conflicts among instructions at up to twelve privilege levels. A response must satisfy all winning instructions and any applicable coding tests. | Johns Hopkins University and collaborators | Following precise instructions | University | Not in cards | GPT-5.4 60.9% | Apr 2026 | Jul 2026 | ||
| MBABench 2.0 Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar. | Columbia University (Namkoong Lab), hosted by Snorkel AI | Finance | Expert | Not in cards | GPT-6 Astra 15.8% | Oct 2026 | Sep 2026 | ||
| MBABench v1 Replaced in Oct 2026 | Columbia University (Namkoong Lab) | Not in cards | ChatGPT agent 31.2 | May 2026 | Mar 2026 | ||||
| MDPBench Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage. | Huazhong University of Science and Technology (MultimodalOCR group) | Documents, charts and text in images | Everyday | Not in cards | | Mar 2026 | Aug 2026 | ||
| MedHELM Models run a suite of medical tasks chosen by clinicians across five categories, from clinical decision support and note writing to patient messages and administration. | Stanford CRFM (HELM) + Stanford Medicine | Medicine | Expert | Not in cards | GPT-5 70.3 | Feb 2025 | Aug 2025 | ||
| MoReBench Models explain decisions in moral dilemmas. A judge scores their reasoning against criteria written by moral philosophers. | MoReBench team with Scale AI | Deception and alignment | Everyday | Not in cards | | Oct 2025 | Sep 2025 | ||
| MoReBench-Theory Models reason about moral decisions under five ethical frameworks. A judge checks their reasoning against framework-specific criteria. | MoReBench team with Scale AI | Deception and alignment | University | Not in cards | GPT-5 79.1 | Oct 2025 | Sep 2025 | ||
| New Synonym Benchmark Models choose the one Swedish word among five options that does not mean the same thing as the target word. | Drasko Markovic | Reading in many languages | Everyday | Not in cards | | Apr 2026 | Aug 2026 | ||
| NPCBench Models follow patients with nasopharyngeal carcinoma through diagnosis and care. The main patient-level measure requires correct decisions at successive stages. | Pengkai Wang and colleagues | Medicine | Expert | Not in cards | Claude Sonnet 4.6 23% | Nov 2026 | Apr 2026 | ||
| OlymMATH Models solve Olympiad-level mathematics problems in English or Chinese. The evaluator checks final answers and reports pass@1 and consistency across repeated samples. | Renmin University of China | Competition problems with answers | Competition | Not in cards | | Mar 2025 | Apr 2025 | ||
| P3B3 Models hold conversations in Portuguese under three language instructions. The judge measures how closely replies follow European or Brazilian usage. | NOVA University of Lisbon | Chat and instructions in many languages | Everyday | Not in cards | | Mar 2026 | Apr 2026 | ||
| ParlaSent Models classify the sentiment of parliamentary sentences as positive, neutral, or negative. Two additional test sets contain 2,600 English and 2,600 South Slavic sentences. | Michal Mochtak and the ParlaSent team | Reading in many languages | Everyday | Not in cards | GPT-5 0.77 | Sep 2023 | Sep 2025 | ||
| PhysVista Vision-language models perceive, reason about and judge whether videos follow real-world physics. The benchmark also tests if they can judge the physical realism of generated videos. | University of Science and Technology of China | Physical prediction | Everyday | Not in cards | GPT-5.2 85% | Sep 2026 | Sep 2026 | ||
| Polaris-Bench Models solve the same 53 visual reasoning tasks in Cartesian and polar layouts. The benchmark compares accuracy across the paired layouts. | Google DeepMind | Spatial reasoning and counting | School | Not in cards | GPT-5.2 39.2% | May 2026 | Apr 2026 | ||
| Qwen-Image-Bench Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images. | Qwen (Alibaba) | Video generation | Everyday | Not in cards | GPT-Image-2 64.7 | May 2026 | Feb 2026 | ||
| RBench-M Models answer 665 graduate-level multimodal reasoning questions in English and Chinese. Top-1 accuracy compares selected answers with the answer key. | Meng-Hao Guo and collaborators | General visual questions | Expert | Not in cards | GPT-5 69.9% | May 2025 | Oct 2025 | ||
| REAL Agents complete 112 tasks on replicas of 11 websites. State checks grade actions, while a model judge checks retrieved answers against references. | AGI, Inc. | Web browsing | Everyday | Not in cards | Claude 3.7 Sonnet 41.1% | Apr 2025 | May 2025 | ||
| RealmPathology Models interpret real anatomic pathology reports and must not claim a diagnosis, stage or treatment the report does not support. Rubrics have 772 positive and 299 penalty criteria. | micro1 | Medicine | Expert | Not in cards | Claude Fable 5.1 78% | Oct 2026 | Oct 2026 | ||
| Senior Engineer Benchmark (Every Checks) Agents repair Proof from the March 17 outage snapshot. Every reports the percentage of 48 checks each solution passes. | Every | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 85% | Oct 2026 | Oct 2026 | ||
| Senior Engineer Benchmark 1.0 Replaced in Oct 2026 | Not in cards | Claude Fable 5 91.0 | Jun 2026 | Sep 2026 | |||||
| SportsEval Models translate 300 English passages about five sports into Icelandic. The scorer checks the Icelandic terms. Humans also judged 37 passages per system at WMT25. | Árni Magnússon Institute for Icelandic Studies | Translation | University | Not in cards | | Jun 2025 | Jun 2025 | ||
| StockBench Agents trade stocks over a fixed evaluation window. StockBench ranks models by the mean of final return, maximum drawdown and Sortino ratio. The score is the board's rank, so a lower score is better. | StockBench authors | Finance | Expert | Not in cards | Kimi K2 1.00 | Oct 2025 | Sep 2025 | ||
| StoreBench Agents manage a simulated apparel store in Medusa. They order inventory, change prices, ship purchases, and process refunds as demand changes. | AfterQuery | Running a business | University | Not in cards | Claude Fable 5.2 0.83 | Oct 2026 | Nov 2026 | ||
| Suicide Memory Bench AI systems, each a model in its own app or coding harness, talk with 128 fictional personas over 29 chats. Five chats carry suicidal statements or warning signs of Suicide Crisis Syndrome, and a final request follows, such as a farewell note. Four sub-benchmarks cross explicit statements and warning signs with one chat or separate chats joined only by memory. A model judge rates each reply on a 0 to 4 safety scale. | Prosaic Intelligence | Wellbeing and sycophancy | Everyday | Not in cards | | Nov 2026 | Oct 2026 | ||
| SuperCLUE-Hard Models follow precise instructions, solve reasoning problems and plan tool calls in Chinese. The report gives separate scores for each ability. | SuperCLUE | Reasoning in many languages | University | Not in cards | o1 76.7 | Dec 2024 | Mar 2025 | ||
| SuperCLUE-LongContext Models retrieve information across multiple turns with long Chinese context. | SuperCLUE | Finding facts in long inputs | School | Not in cards | | Nov 2026 | |||
| SuperCLUE-WebCoding Models implement web interfaces from specifications. | SuperCLUE | Web and app front ends | University | Not in cards | | Dec 2025 | |||
| SveaKnow Culture Models give short answers to factual questions about Swedish history, traditions, media, education, and culture. | Drasko Markovic | Knowledge in many languages | School | Not in cards | | Apr 2026 | Aug 2026 | ||
| TestPrism Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one. | Nanjing University (NJU-LINK Lab) | Function-level problems | Expert | Not in cards | Kimi K3 33.3% | Oct 2026 | Sep 2026 | ||
| Time Horizon Index: KSP Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions. | Vals AI | Video games | Expert | Not in cards | Claude Opus 5.5 91.3% | Jul 2026 | Sep 2026 | ||
| tr_gender_bias Models translate 200 Turkish passages into English. Set B scores pronoun accuracy on 82 passages where a name or kinship term identifies gender. | Boğaziçi University | Translation | Everyday | Not in cards | | Jun 2026 | May 2026 | ||
| TrafficImag Models edit roadside traffic videos to show a different sequence of events. The evaluator checks routes, interactions, and preservation of the surrounding scene. | Xiangyu Li and colleagues | Video generation | Everyday | Not in cards | GPT-Image-2 90.0 | Sep 2026 | Sep 2025 | ||
| Vals Smith: apache/spark Agents modify apache/spark to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 84.5% | Aug 2026 | Jul 2026 | ||
| Vals Smith: astral-sh/uv Agents modify astral-sh/uv to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Fable 5 66% | Aug 2026 | Jul 2026 | ||
| Vals Smith: jax-ml/jax Agents modify jax-ml/jax to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 79.6% | Aug 2026 | Jul 2026 | ||
| Vals Smith: oven-sh/bun Agents modify oven-sh/bun to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Kimi K3 54.2% | Aug 2026 | Jul 2026 | ||
| Vals Smith: palantir/blueprint Agents modify palantir/blueprint to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Fable 5 93.2% | Aug 2026 | Jul 2026 | ||
| Vals Smith: sgl-project/sglang Agents modify sgl-project/sglang to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Terra 78% | Aug 2026 | Jul 2026 | ||
| Vals Smith: vercel/next.js Agents modify vercel/next.js to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 74.3% | Aug 2026 | Jul 2026 | ||
| Which AI Model Knows Israel Best? Models answer Hebrew questions about everyday life in Israel. The organizers check each answer against an official source and count refusals separately. | Skills IL | Knowledge in many languages | Everyday | Not in cards | Grok 4.7 99.3% | Oct 2026 | Oct 2026 | ||
| WitnessBench Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels. | Fleet AI | Puzzles and chess | Expert | Not in cards | Claude Opus 5.5 46.5% | Sep 2026 | Sep 2026 | ||
| WMT25 EEA Terminology Models translate English regulatory sentences into Icelandic. The scorer checks whether they use the official Icelandic terms from the European Economic Area agreement. | Árni Magnússon Institute for Icelandic Studies and University of Iceland | Translation | University | Not in cards | | Jun 2025 | Jun 2025 | ||
| Agent Initiative Benchmark Agents inspect workplace situations and decide whether to act without an explicit task. Two model judges score responses to 50 situations from 0 to 10. | Deepnote | Simulated companies | Expert | Not in cards | GPT-6 Astra 8.97 | Nov 2026 | |||
| AGI-Eval Community Text to Image Models generate images from text. The board reports objective, subjective and combined scores. | AGI-Eval (Meituan) | Image generation and editing | Everyday | Not in cards | | Feb 2025 | Mar 2025 | ||
| Aider Refactoring Benchmark Models refactor large Python methods while preserving their implementation. The benchmark checks whether edits complete correctly. | Aider | Fixing real repositories | University | Not in cards | Claude 3.5 Sonnet (October 2024) 92.1% | Dec 2023 | Oct 2024 | ||
| ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. | Tsinghua University and collaborators | Simulated robots | Everyday | Not in cards | | Nov 2026 | Feb 2026 | ||
| AnvilBench Models write compute kernels for Apple Silicon. Tasks cover operations used in attention, decoding, and audio processing. | Vikram Tiwari | GPU kernels | Expert | Not in cards | | Oct 2026 | Nov 2026 | ||
| AppWorld Test-Challenge Agents complete 417 app tasks that require Amazon or Gmail APIs held out from training. The score is the percentage passing every task test. | Stony Brook University and Allen Institute for AI | Apps and MCP servers | Everyday | Not in cards | GLM-5.3-Flash 92.8% | Jul 2024 | Aug 2026 | ||
| AppWorld Test-Normal Agents complete 168 everyday app tasks from familiar applications by calling APIs through code. The score is the percentage passing every task test. | Stony Brook University and Allen Institute for AI | Apps and MCP servers | Everyday | Not in cards | GLM-5.3 94% | Jul 2024 | Aug 2026 | ||
| ASI-Bench AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge. | Apexin AI | Research science | Research frontier | Not in cards | GPT-5.6 Sol 55.8 | Aug 2026 | Aug 2026 | ||
| AutoResearchExam Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed. | Bespoke Labs | Machine learning engineering | Research frontier | Not in cards | Claude Opus 5.5 0.61 | Sep 2026 | Sep 2026 | ||
| BenGER Models write German legal analyses for 596 exam-style cases and 531 doctrinal reasoning tasks. A rubric judge scores legal conclusions, reasoning, and presentation against expert references. Case scores map to the German 0-to-18 grading scale, with four points required to pass. | Technical University of Munich, Ludwig Maximilian University of Munich, University of Konstanz, and Saarland University | Law | Expert | Not in cards | | May 2026 | Apr 2026 | ||
| Bio-PPI Risk: Serous vs Endometrioid Models estimate a cancer subtype probability from tumor protein measurements and protein interactions. The evaluation measures discrimination and probability accuracy. | Neha | Medicine | Expert | Not in cards | Claude Opus 5.5 0.22 | Nov 2026 | Oct 2026 | ||
| BioSimBench Models predict which genes change after drugs, gene knockdowns, and immune signals across four prediction tracks containing 60, 60, 60, and 27 cases. The overall score averages perturbation-specific F1 across those tracks on a zero-to-one scale. | CellType | Genomics and life sciences | Expert | Not in cards | | Nov 2026 | Oct 2026 | ||
| CAD Arena Agents rebuild licensed engineering drawings as editable CAD parts through the scripting APIs of NX, SolidWorks, Onshape, Fusion or Build123d. A verifier scores geometric match. | Normal Research | CAD models | Expert | Not in cards | | Sep 2026 | Sep 2026 | ||
| CAPR-am Models read an Amharic scenario and choose the best matching proverb from four options. The authors score exact choices. | Haverford College | Reasoning in many languages | Everyday | Not in cards | | Jul 2026 | Aug 2025 | ||
| CAPR-ar Models read an Arabic scenario and choose the best matching proverb from four options. The authors score exact choices. | Haverford College | Reasoning in many languages | Everyday | Not in cards | | Jul 2026 | Aug 2025 | ||
| CAPR-ja Models read a Japanese scenario and choose the best matching proverb from four options. The authors score exact choices. | Haverford College | Reasoning in many languages | Everyday | Not in cards | | Jul 2026 | Aug 2025 | ||
| CAR-bench Agents act as an in-car voice assistant that talks with an LLM-simulated driver and uses 58 connected tools for navigation, productivity, charging and vehicle control under set policies. Besides ordinary tasks, hallucination tasks remove a needed tool or fact, and disambiguation tasks need a clarifying question or a lookup. Scores reward consistent success across repeated runs. | BMW Group and University of Augsburg | General assistants | Everyday | Not in cards | GPT-5.6 Sol 70% | Jan 2026 | Jul 2026 | ||
| CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. | The University of Hong Kong | Video understanding | Expert | Not in cards | Claude Opus 5.5 76.6 | Oct 2026 | Sep 2026 | ||
| Claw-Eval-Live Agents complete 105 workflow tasks in 17 families drawn from live ClawHub workflow signals, scored on both the workflow and the final workspace state. | Claw-Eval-Live team (Li, Tang et al.) | General assistants | University | Not in cards | Claude Opus 4.6 66.7% | Apr 2026 | Mar 2026 | ||
| Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. | Xiaokang Ye and colleagues | Image generation and editing | University | Not in cards | GPT-6 Astra 0.62 | Sep 2026 | Sep 2026 | ||
| ComplexBench Models answer 1,150 hand-written Chinese instructions that compose several constraints by chaining, selection and nesting. LLM evaluators with rules check each constraint, and scores follow the dependency structure. | Tsinghua University (CoAI) and Zhipu AI | Following precise instructions | Everyday | Not in cards | GPT-4 Turbo 80% | Jul 2024 | Apr 2024 | ||
| CounterBench (1,200 questions) Models answer counterfactual questions using the stated causal rules. This release adds 200 backdoor questions to the original 1,000. | Rutgers University and Case Western Reserve University | Language reasoning | University | Not in cards | | Aug 2025 | Dec 2024 | ||
| CounterBench 1,000 questions Replaced in Aug 2025 | Not in cards | | Feb 2025 | Dec 2024 | |||||
| Coverage Cat AI Insurance Benchmark (fictional insurance tasks) Models read fictional policy documents, find missing facts, and complete insurance applications. The scorer checks source quotes, required fields, and authorized actions. | Coverage Cat | Customer service | University | Not in cards | GPT-6 Astra 100% | Nov 2026 | Oct 2026 | ||
| Credit Firewall Benchmark Models decide whether evidence still authorizes a credit decision after roles, targets, timing, or conversation context change. | Christine Classy — AI’s Favorite™ | Deception and alignment | University | Not in cards | Claude Haiku 4.5 100% | Oct 2026 | Sep 2026 | ||
| CriticEval Models critique responses across four dimensions in nine task scenarios, giving both scalar scores and written critiques. GPT-4 grades written critiques against a large set of annotated reference critiques. | OpenCompass (Shanghai AI Laboratory) and collaborators | Overall chat quality | University | Not in cards | GPT-4 Turbo 7.86 | Feb 2024 | Jan 2024 | ||
| CypherBench Models translate questions about Wikidata into Cypher queries. The scorer executes each query and compares its result with the expected answer. | Megagon Labs and collaborators | Data science and SQL | University | Not in cards | Claude 3.5 Sonnet 61.6% | Dec 2024 | Sep 2024 | ||
| DeskCraft Desktop agents complete 538 Ubuntu tasks, including 152 tasks with later user clarifications or interruptions. Programmatic checks inspect final application state, saved files, and exported artifacts. | Zhejiang University, Tsinghua University, Tencent and others | Desktop apps | Expert | Not in cards | Kimi K2.6 29.8% | Jun 2026 | Apr 2026 | ||
| DrivingBench v2 Vision-language models drive a real Tesla through two unseen cone courses with steering, accelerator and brake tools. The score is the furthest progress over three attempts in one conversation, averaged over six trials. | DrivingBench (Ramabadran, Mahns and Gessler) | Real robots | Everyday | Not in cards | | Nov 2026 | Nov 2026 | ||
| DrivingBench v1 Replaced in Nov 2026 | Not in cards | GPT-6 Astra 100% | Sep 2026 | Sep 2026 | |||||
| EGI-BENCH Agents work across eight enterprise evaluations. The suite combines their results into six scores covering activities such as planning, retrieving evidence, and using tools. | Salesforce AI Research | Simulated companies | University | Not in cards | GPT-5.5 65.6 | Oct 2026 | Apr 2026 | ||
| Emergent Price-Fixing by LLM Auction Agents Models trade in simulated auctions and communicate with other participants. A judge examines game logs for evidence of price fixing. | Lech Mazur | Deception and alignment | Everyday | Not in cards | Grok 3 19% | Jul 2025 | Jul 2025 | ||
| Enterprise-Bench L1-L2 v2 Agents answer 42 operational questions using six enterprise services. A GPT-6 Luna judge evaluates the final answer against task criteria. | DevRev | Apps and MCP servers | University | Not in cards | GPT-6.1 Sol 65.7% | Nov 2026 | Sep 2026 | ||
| Enterprise-Bench L1-L2 Replaced in Nov 2026 | Not in cards | Claude Opus 5 82.9% | Jul 2026 | Sep 2026 | |||||
| Exact Words Models copy a requested sentence exactly from a document. The grader checks sentence boundaries and whether the model preserves planted typos. | Saharsh Tibrewala | Faithful to sources | School | Not in cards | GLM-5 100% | Oct 2026 | Oct 2026 | ||
| FDE-Gym Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1. | DeepLethe (with 20+ contributing institutions) | Building whole projects | Expert | Not in cards | Claude Opus 5.5 0.56 | Oct 2026 | Sep 2026 | ||
| GeoGuessBench (Nutlope) Models place a pin after seeing one street photo. Each game has five rounds, and closer guesses earn more points. | Nutlope | Spatial reasoning and counting | Everyday | Not in cards | Claude Opus 5.5 23412.0 | Oct 2026 | Sep 2026 | ||
| Glianorex Models answer multiple-choice questions about fictional medical material in English and French. The test isolates question-answering patterns from real medical knowledge. | Maxime Griot and colleagues | Language reasoning | School | Not in cards | GPT-4o 71.5% | Jun 2024 | Mar 2025 | ||
| HalluWorld Language models are evaluated on hallucinations across controlled reference worlds spanning gridworlds, chess, and realistic terminal environments. Each environment has a fully specified ground truth, allowing automatic evaluation of whether model claims are correct. Questions test perception, memory, causal reasoning, uncertainty/abstention, and compound reasoning under varying levels of observability, complexity, and conflicting information. | Other | Not in cards | Claude Fable 5.1 19.2% | Sep 2026 | |||||
| InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. | Meta + Duke University | Simulated robots | Everyday | Not in cards | Claude Fable 5.1 41.7% | Oct 2026 | Sep 2026 | ||
| JFinBench Models answer questions from six Japanese certification exams in accounting, tax, financial planning, and banking. The authors report scores by exam. | Sakana AI | Finance | University | Not in cards | GPT-6 Astra 97.3% | Nov 2026 | Oct 2026 | ||
| KORGym Models play 51 text and visual games designed to test interactive reasoning. | KORGym authors | Puzzles and chess | Everyday | Not in cards | o3-mini 0.82 | May 2025 | Apr 2025 | ||
| Loss Run Extraction (Bevaya) Models extract claims and policy tables from insurance loss reports into JSON. The main metric counts documents where every field matches the reference. | Bevaya Labs | Documents, charts and text in images | University | Not in cards | Claude Fable 5 85.3% | Aug 2026 | Jul 2026 | ||
| Mastermind (Prime Intellect) Models guess a hidden sequence and receive feedback after each turn. The default evaluation uses 50 games and rewards success, speed, and deductions. | Prime Intellect | Puzzles and chess | Everyday | Not in cards | GPT-5 1.62 | Oct 2025 | Sep 2025 | ||
| MathIF Reasoning models solve 420 math problems from GSM8K, MATH-500, Minerva, Olympiad and AIME while obeying added verifiable instructions. Code checks the instructions and the answer. | Renmin University of China, Shanghai AI Laboratory and CUHK | Following precise instructions | School | Not in cards | | May 2025 | Apr 2025 | ||
| MathTutorBench Models solve math problems, diagnose student mistakes, ask Socratic questions, and give instructional support. Automatic answer checks, F1, and text similarity score the constrained tasks. A trained pedagogical reward model compares tutoring responses with teacher examples. | ETH Zurich and TU Darmstadt | Word problems | School | Not in cards | | Feb 2025 | Jul 2026 | ||
| MedPhysBench Models calculate doses and check medical physics worksheets for planted errors. A grader checks the units and accepts values within a specified tolerance. | Omar Nusrat | Medicine | Expert | Not in cards | GPT-6 Astra 100% | Nov 2026 | Sep 2026 | ||
| MedQA-Followup Models answer medical licensing questions and then respond to follow-ups that challenge their answers. Researchers compare accuracy before and after each intervention. | Dynamo AI and collaborators | Medicine | Expert | Not in cards | GPT-5 mini 0% | Oct 2025 | Nov 2025 | ||
| MineAmongUs Vision-language agents play an embodied Among Us game with verbal and nonverbal deception. | MineAmongUs authors | Multi-agent games | Everyday | Not in cards | | Aug 2026 | Apr 2026 | ||
| Monthly-SWEBench Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1. | UniPat AI and xbench (HongShan) | Fixing real repositories | Expert | Not in cards | GPT-5.5 76% | Mar 2026 | Jun 2026 | ||
| NEST Models interpret children's gaze, gestures, and physical contact in social videos. The benchmark checks classification and choice questions and separately grades written descriptions. | Stanford University and collaborators | Video understanding | Everyday | Not in cards | | Nov 2026 | Mar 2026 | ||
| OmniScore-Bench Models read sheet music, locate symbols, and transcribe passages. The evaluator checks their responses against annotations for the relevant part of the score. | Tsinghua University and collaborators | Documents, charts and text in images | University | Not in cards | GPT-6.1 Sol 91.0 | Nov 2026 | Oct 2026 | ||
| Paragraph Instruction Following Models write a single plain text paragraph under a word count constraint. The tasks check length, format, language, and relevance across 500 prompts. | Łukasz Wodniakowski | Following precise instructions | School | Not in cards | GPT-5.5 0.99 | Oct 2026 | Sep 2026 | ||
| ParliamentBench Models play Secret Hitler while deceiving, persuading, and reasoning under hidden roles. | ParliamentBench authors | Multi-agent games | Everyday | Not in cards | GPT-5.4 81% | Jul 2026 | Mar 2026 | ||
| PDAGENT-BENCH Models answer chip design questions, diagnose failures, interpret reports, and write tool commands. Rules grade five categories and model judges grade three. | Qiufeng Li and colleagues | Electronics | Expert | Not in cards | GPT-5.5 65.1% | Jun 2026 | Sep 2026 | ||
| PlanningBench Models solve generated planning problems that coordinate goals, constraints and resources. Instances are scalable and come with verifiers. | Tencent Hunyuan | Language reasoning | Expert | Not in cards | GPT-5.4 63.2% | May 2026 | Apr 2026 | ||
| PowerCodeBench Models translate operators' requests into Python programs for pandapower. The evaluator runs each program and compares its numerical or Boolean output with the reference answer. | University of Exeter | Scientific code | University | Not in cards | | May 2026 | Apr 2026 | ||
| Prompt-to-Bench-16 Models write OpenSCAD programs for laboratory fixtures from measured dimensions. They receive feedback after each failed attempt. The grader checks the rendered geometry. | S. S. Cocioba, Binomica Labs | CAD models | University | Not in cards | | Oct 2026 | Aug 2026 | ||
| PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. | Scale AI and collaborators | Medicine | Expert | Not in cards | | Jun 2026 | Apr 2026 | ||
| RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. | Robocurve PBC | Real robots | Everyday | Not in cards | GPT-6.2 Astra 65.7% | Sep 2026 | Nov 2026 | ||
| Rule2DRC Models write KLayout scripts from chip design rules. The evaluator checks whether each script flags the correct violations on every test layout. | Seoul National University and collaborators | Electronics | Expert | Not in cards | Kimi K3 67.9% | May 2026 | Jul 2026 | ||
| SAHM Models answer Arabic questions on financial standards, Islamic finance, and accounting. The authors report multiple-choice accuracy and grade open-ended answers separately. | Rania Elbadry and colleagues | Finance | University | Not in cards | | Apr 2026 | Dec 2025 | ||
| Samskritam Bench v1 Models split entire Sanskrit verses into words, parse inflected words, scan verse meter, and generate rare verb forms. Each verse requires every graded part to match. | Sriram Subramanian, Samskritam.ai | Reading in many languages | University | Not in cards | Claude Opus 5.5 89.4% | Oct 2026 | Sep 2026 | ||
| Samskritam Bench v0 Replaced in Oct 2026 | Not in cards | Claude Opus 5.5 99.8% | Oct 2026 | Sep 2026 | |||||
| ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. | UniPat AI and Elorian | Spatial reasoning and counting | Expert | Not in cards | GPT-6 Astra 0.53 | Oct 2026 | Nov 2026 | ||
| SciPredict Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge. | Scale AI | Research science | Expert | Not in cards | | Jan 2026 | Dec 2025 | ||
| SimBench (PyChrono) Models write and revise PyChrono simulation programs over three turns. A model grades each program using the reference solution and API documentation. | University of Wisconsin–Madison Simulation-Based Engineering Laboratory | Scientific code | University | Not in cards | Claude Sonnet 4 49.0 | Aug 2024 | Jun 2025 | ||
| Slay the Spire Bench (alavico) Models choose an action from the text state of Slay the Spire. The campaign report measures floors reached and wins at each difficulty. | alavico | Video games | Everyday | Not in cards | Claude Sonnet 4.6 51.0 | Jun 2026 | Jul 2026 | ||
| Snowflake Text-to-SQL HARD Models generate SQL for difficult questions about unfamiliar enterprise schemas. Snowflake compares models by query accuracy. | Snowflake | Data science and SQL | Expert | Not in cards | | May 2026 | Apr 2026 | ||
| SpireClimb Agents play Slay the Spire 1 and 2 at max difficulty through a computer-use harness on fixed seeds. Score is the mean win rate. | Epoch AI | Video games | Expert | Not in cards | GPT-6.2 Astra 41.3% | Oct 2026 | Nov 2026 | ||
| SpireClimb StS1 A20 Heart Agents play Slay the Spire 1 through its graphical interface at Ascension 20 and must defeat the Heart. Scored by win rate over fixed seeds. | Epoch AI | Video games | Expert | Not in cards | GPT-6 Astra 33.3% | Oct 2026 | Nov 2026 | ||
| SpireClimb StS2 A10 Agents play Slay the Spire 2 at Ascension 10 through screenshots and mouse and keyboard, 15 runs per character. Scoring is the win rate. | Epoch AI | Video games | Everyday | Not in cards | GPT-6.2 Astra 49.3% | Oct 2026 | Nov 2026 | ||
| STELLA-Bench Models respond to simulated users over multiple turns. Judges measure the shares of conversations with harmful or unhelpful responses. | Atella AI | Wellbeing and sycophancy | Everyday | Not in cards | GPT-5.6 Terra 0% | Sep 2026 | |||
| SuperCLUE-CPIF Models follow precise instructions written in Chinese. | SuperCLUE | Following precise instructions | Everyday | Not in cards | GPT-5.1 83.6 | Nov 2025 | |||
| SuperCLUE-SWE with Claude Code Models solve Chinese software engineering tasks in the Claude Code harness. | SuperCLUE | Fixing real repositories | Expert | Not in cards | Kimi K3 75.8 | Aug 2026 | |||
| SuperCLUE-Writing Models write original Chinese prose. | SuperCLUE | Creative writing | University | Not in cards | Claude Opus 4.7 88.0 | May 2026 | |||
| TableBench Models answer 886 questions about real-world tables across 18 fields in four groups: fact checking, numerical reasoning, data analysis and visualization. Fact and numeric answers are scored by exact match, analysis answers by ROUGE-L, and chart code by pass@1. | Beihang University, M-A-P and collaborators | Language reasoning | University | Not in cards | o4-mini 61.7% | Aug 2024 | Aug 2025 | ||
| TimelineBench Agents turn raw footage and a client brief into a finished video. The edit must pass delivery checks, meet the brief, and clear a quality threshold. | TensorTest | Tasks from real jobs | University | Not in cards | Claude Opus 5.5 32.1% | Sep 2026 | Sep 2026 | ||
| TreasuryQA Bench Agents search scanned Treasury volumes and read tables to find or calculate a number. A script checks whether the answer falls within the accepted range. | Veris AI and Reducto | Documents and spreadsheets | School | Not in cards | | Oct 2026 | Nov 2026 | ||
| trivial-prompt-bench Models answer a short request through a terminal agent. Quesma adds the API charge to the estimated cost of the developer's waiting time. | Quesma | Other | Everyday | Not in cards | GPT-5.4 Mini $0 | Jul 2026 | Jun 2026 | ||
| TUA-Bench Agents use a terminal for general computer tasks beyond coding, such as file, system and everyday productivity work. | Meta | Terminal tasks | Everyday | Not in cards | Claude Opus 4.8 65.8% | Jun 2026 | Jun 2026 | ||
| Veris FinAgent Arena v1.1 Agents investigate bank alerts, read Treasury scans, and complete research requests. The index combines accuracy and repeatability across three benchmarks, then applies a cost penalty. | Veris AI | Composite work indices | University | Not in cards | | Nov 2026 | Nov 2026 | ||
| WearableQA Models answer 4,084 ten-option questions using longitudinal wearable measurements, blood biomarkers, and demographics from 200 users. Questions test computations and physiological interpretation across single or multiple signals. Accuracy compares the selected option with the answer key. | Meta, KAIST, and Korea University | Medicine | Expert | Not in cards | | Sep 2026 | Apr 2026 | ||
| Web Context (Nimble) Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages. | Nimble | Deep research | Expert | Not in cards | GPT-5.6 Sol 80% | Oct 2026 | Oct 2026 | ||
| WellnessBench Models answer young people asking about eating, sexual safety, substances and self harm, in clinician-written single-turn and pushback dialogues. A pass must be both safe and helpful. | micro1 | Wellbeing and sycophancy | Everyday | Not in cards | GPT-6.1 Sol 66% | Oct 2026 | Oct 2026 | ||
| Who Are You? LLM Self-Identity Models answer twelve phrasings of questions about their identity and maker. The grader compares their answers with the model Kaggle ran. | J Gonn6235 | Short factual questions | Everyday | Not in cards | gpt-oss-120b 100% | Oct 2026 | Sep 2026 | ||
| AECV-Bench (2026 release) Models count four types of floor plan elements and answer questions about engineering drawings. Counting uses exact matches. A model grades the drawing answers. | AECFoundry | Documents, charts and text in images | University | Not in cards | | Dec 2025 | Jun 2026 | ||
| Agentick Agents solve procedurally generated gridworld tasks involving navigation, planning, reasoning, memory, generalization, and coordination. Scores normalize task performance between random and oracle reference policies. | Roger Creus Castanyer, Pablo Samuel Castro, and Glen Berseth | Puzzles and chess | Everyday | Not in cards | GPT-6 Astra 0.63 | May 2026 | Sep 2026 | ||
| AgReason Models answer 100 farm management questions using local growing conditions. A judge compares individual claims with expert answers. A question passes at statement-level F1 of 0.80. | Iowa State University and New York University | Science questions | University | Not in cards | | May 2025 | Sep 2025 | ||
| AI Engineering Lab model creation Models generate Exudyn simulation code for 35 mechanical systems. The evaluator compares numerical outputs with reference simulations across ten parameter settings per system. | University of Innsbruck | Scientific code | University | Not in cards | | Apr 2025 | Jan 2025 | ||
| AMemGym Assistants hold long simulated conversations with users whose details and preferences change over time. Simulated users react on policy, and the test checks whether the assistant remembers and applies the right facts. | AGI-Eval (Meituan) | Learning from the context | Everyday | Not in cards | | Jan 2026 | Dec 2025 | ||
| Arena-T2I Hard Models generate images from detailed user prompts. A judge checks whether the images satisfy individual requirements and their dependencies. | Arena | Image generation and editing | University | Not in cards | | Jun 2026 | |||
| ATLAS-Finance v1.1.0 Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level. | Handshake AI Research | Finance | Expert | Not in cards | Claude Opus 5 10.7% | Oct 2026 | Sep 2026 | ||
| ATLAS-Finance 1.0.0 Replaced in Oct 2026 | Not in cards | Sep 2026 | |||||||
| AutoFactory-Bench Agents write control programs for simulated factory cells. A run passes when the program completes every operating condition without a safety violation. | AutoFactory-Bench with Dongguk University and MOAI technologies | Electronics | University | Not in cards | GPT-6 Astra 72.2% | Oct 2026 | Oct 2026 | ||
| Basic reasoning with letters Models count and compare letters within words. The tasks include counting a letter across weekday names and finding words with equal character frequencies. | Navneet Prabhat | Language reasoning | School | Not in cards | GLM-5 100% | Oct 2026 | Sep 2026 | ||
| BIG-Bench Extra Hard Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. | Google DeepMind | Language reasoning | University | Not in cards | o3-mini 44.8% | Feb 2025 | Mar 2025 | ||
| BTF-3 Agents forecast already-resolved binary and numeric questions while researching a frozen snapshot of the web. Scoring pools Brier and ranked probability scores. | FutureSearch | Real-world events | Expert | Not in cards | Claude Opus 5 0.12 | Jun 2026 | Sep 2026 | ||
| BTF-2 Replaced in Jun 2026 | Not in cards | Claude Opus 4.6 0.13 | Apr 2026 | Mar 2026 | |||||
| Bench to the Future v1 Replaced in Apr 2026 | Not in cards | Jun 2025 | |||||||
| CaBBQ Models answer Catalan questions about people in ambiguous and unambiguous situations. The test measures whether answers follow social stereotypes when evidence is insufficient. | Valle Ruiz-Fernández and colleagues | Bias, fairness and politics | Everyday | Not in cards | | Jul 2025 | Mar 2025 | ||
| CASE-Bench Models decide whether a request is safe in a given context. The same request appears in two contexts with different human judgments. | Guangzhi Sun and colleagues | Refusing harmful requests | Everyday | Not in cards | Claude 3.5 Sonnet 90.1% | Jan 2025 | Aug 2024 | ||
| Continual Learning Exam Agents work through 120 professional tasks, each a ladder of five levels that must be passed in order. Later levels reuse files, tools and lessons from earlier ones. An evaluation agent grades against expert rubrics. | University College London (Jun Wang) + Imperial College London + University of Liverpool | Learning from the context | Expert | Not in cards | Claude Opus 5 47.7% | Oct 2026 | Nov 2026 | ||
| COREBench (Conversational Reasoning) Models listen to a conversation with several speakers and answer a question that can require information from multiple turns. | Stanford CRFM and the AHELM team | Audio understanding and voice | Everyday | Not in cards | | Jun 2025 | May 2025 | ||
| CoreCodeBench Models complete over 1,500 repository-level Python tasks covering development, bug fixing and test-driven work, with single-function and multi-function variants. The CorePipe pipeline builds tasks from repositories at set difficulty levels. | AGI-Eval (Meituan) | Fixing real repositories | Expert | Not in cards | | Jul 2025 | Dec 2025 | ||
| Crazyrouter AI Trading Arena Models buy, sell, or hold three cryptocurrencies every five minutes using simulated accounts. The grader values portfolios at live quotes and deducts trading fees. | Crazyrouter | Finance | University | Not in cards | GPT-6.1 Sol -7.45% | Nov 2026 | Oct 2026 | ||
| CrossPoint-Bench Models see two views of a scene and must find the point in the second view that matches a point in the first. Tasks go from perception and reasoning to exact coordinate prediction. | Institute of Automation, Chinese Academy of Sciences (Wang, Ji et al.) | Spatial reasoning and counting | Everyday | Not in cards | | Dec 2025 | Oct 2025 | ||
| CyberMetric Models answer multiple-choice cybersecurity knowledge questions drawn from standards, papers and RFCs. Sets of 80, 500, 2,000 and 10,000 questions exist. | Technology Innovation Institute (Abu Dhabi) | Capture-the-flag challenges | Expert | Not in cards | | Feb 2024 | Apr 2024 | ||
| DashBench Models explain rare events in driving videos, including near misses and collisions. The authors compare 25 video models on 1,709 clips. | Korea Advanced Institute of Science and Technology | Video understanding | Everyday | Not in cards | GPT-5 34.8% | Nov 2026 | Dec 2025 | ||
| DeputyBench Agents handle business requests under paired permission settings. The evaluator checks whether they act when authorized and seek evidence or refrain when authorization is missing. | University of Southern California | Simulated companies | University | Not in cards | | Nov 2026 | Nov 2026 | ||
| Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. | micro1 | Simulated robots | Expert | Not in cards | Claude Fable 5.2 59.8 | Oct 2026 | Nov 2026 | ||
| EarthRecord Agents analyze multidecade satellite observations to answer land-change questions. Reference checks score answers to 238 questions across 56 sites. | AIEyes team | Lab and data work | Expert | Not in cards | Claude Opus 5.5 85.4% | Nov 2026 | Oct 2026 | ||
| EJBench Models interpret simulation results for battery cooling plates. They match results to test conditions, select evidence for a design decision, and rank operating conditions by temperature. | Gradient Dynamics | Lab and data work | University | Not in cards | GPT-6 Astra 0.71 | Nov 2026 | Oct 2026 | ||
| Encord Egocentric VQA Benchmark Models answer 1,500 multiple-choice questions about actions, objects, space and time in private first-person videos. | Encord Data Labs | General visual questions | Everyday | Not in cards | GPT-6 Astra 76.5% | Nov 2026 | Sep 2026 | ||
| EsBBQ Models answer Spanish questions about people in ambiguous and unambiguous situations. The evaluation measures how social stereotypes affect their choices. | Valle Ruiz-Fernández and colleagues | Bias, fairness and politics | Everyday | Not in cards | | Jul 2025 | Mar 2025 | ||
| EvalDetectBench A judge model reads a transcript and answers whether it came from an evaluation or from real deployment. The test measures how well models detect evaluations and how detectable each benchmark is. | UK AI Security Institute + LASR Labs | Deception and alignment | Expert | Not in cards | Claude Opus 4.6 0.89 | Jun 2026 | Apr 2026 | ||
| Frontier3DBench Agents create editable Blender objects and scenes or executable Unreal Engine worlds across 145 tasks and nine task families. Model judgments and native project checks assess appearance, geometry, requirements, and interactions. The overall score weights object, scene, and world scores by 30%, 30%, and 40%. | Einsia AI (Navers lab) | Spatial reasoning and counting | Expert | Not in cards | GPT-6.1 Sol 66.0 | Nov 2026 | Nov 2026 | ||
| FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. | Apodex AI (FrontierAgent team) | Research science | Expert | Not in cards | GPT-5.6 Sol 20.6% | Aug 2026 | Aug 2026 | ||
| Hard12 Agents solve 12 selected coding tasks from five benchmarks. Hidden tests grade each task. The leaderboard averages three runs per task. | Ibragim Badertdinov | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 83.3% | Nov 2026 | Oct 2026 | ||
| Hedge-Bench Agents analyze supplied company and investment documents for 102 open-ended financial research tasks. A model judge applies rubrics that check whether the reasoning covers the analytical steps identified by investment professionals. | Trata | Finance | Expert | Not in cards | GPT-5.6 Sol 31.5 | Jun 2026 | Sep 2026 | ||
| HelloBench Models generate long answers, summaries, conversations, text completions, and texts from writing instructions. Model judges apply task-specific checklists to the responses. | Beihang University, Shanghai AI Laboratory, Peking University and others | Overall chat quality | University | Not in cards | GPT-4o 48.5 | Sep 2024 | Sep 2024 | ||
| Human Behavior Bench Models predict how specified groups of U.S. adults answer closed survey questions, as a distribution over options. Scored as 1 minus total variation distance. | Rehearsals with Google DeepMind and Kaggle | Real-world events | Expert | Not in cards | | Oct 2026 | Sep 2026 | ||
| IFMTBench Models translate text while following explicit instructions, such as keeping JSON or HTML structure, using a glossary, using provided context and matching a register. It covers seven languages with 4,506 single-constraint and 2,838 multi-constraint items. Deterministic checkers verify some constraints, and a rubric-based model judge scores the rest. | Tencent Hunyuan | Translation | University | Not in cards | | May 2026 | Apr 2026 | ||
| IPTC-test Models choose one of 17 newsroom topics for each article. Human annotators labeled 1,129 articles in Catalan, Croatian, Greek, and Slovenian. | Taja Kuzman and Nikola Ljubešić | Reading in many languages | Everyday | Not in cards | | Nov 2024 | Sep 2025 | ||
| IrokoBench Models answer human-translated knowledge, math and inference questions in 16 African languages. The parts are AfriMMLU, AfriMGSM and AfriXNLI. | Masakhane and collaborators (Adelani et al.) | Knowledge in many languages | School | Not in cards | GPT-4o 59% | Jun 2024 | Mar 2025 | ||
| Judge Bait Models judge pairs of answers. The test varies their order, length, authority claims, and pressure cues to see whether judges prefer the correct one. | Kelechukwu Agbo | Language reasoning | School | Not in cards | Claude Sonnet 5.5 100% | Oct 2026 | Oct 2026 | ||
| LABBench2 LitQA3 Models answer open-response questions whose answers appear once in recent biology papers. The LitQA3 subtask of LABBench2. | Edison Scientific (formerly FutureHouse) | Genomics and life sciences | Expert | Not in cards | GPT-5.2 Pro 85.1% | Feb 2026 | Feb 2026 | ||
| LAB-Bench LitQA2 v1 Replaced in Feb 2026 | FutureHouse | Not in cards | Claude 3 Haiku 27% | Jul 2024 | Jun 2024 | ||||
| LitQA v0 Replaced in Jul 2024 | Research science | Not in cards | Claude 2 40.6% | Dec 2023 | Jul 2023 | ||||
| Labelbox Complex Reasoning Models solve reasoning problems in mathematics, programming, and other domains. Labelbox reports overall and domain scores. | Labelbox | Language reasoning | University | Not in cards | GPT-5 89.6% | Jul 2025 | Aug 2025 | ||
| LawBench Models answer Chinese legal questions across 20 tasks, including statute recall and case prediction. Each task has its own scoring rule. | Zhiwei Fei and colleagues | Law | University | Not in cards | GPT-3.5 Turbo 44.5 | Sep 2023 | Sep 2023 | ||
| Learn2Play Bench Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure. | National University of Singapore | Puzzles and chess | Everyday | Not in cards | Claude Opus 5 81.8 | Oct 2026 | Sep 2026 | ||
| LegalReasonBench Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings. | micro1 | Law | Expert | Not in cards | | Oct 2026 | Nov 2026 | ||
| LLM Persuasion Benchmark Models try to change another model’s stated position over multiple conversation turns. | Lech Mazur | Multi-turn conversation | Everyday | Not in cards | GPT-5.4 1.71 | Mar 2026 | |||
| LocalSearchBench Agents search merchant listings to answer 900 questions about local services across nine Chinese cities. A model judges each answer against a reference. | Meituan, East China Normal University, and collaborators | Questions that need search | Everyday | Not in cards | | Dec 2025 | Dec 2025 | ||
| M5HisDoc Models locate text, recognize characters, and recover reading order on Chinese historical pages. The hard subset distorts the same 4,000 pages as the regular subset. | South China University of Technology | Documents, charts and text in images | University | Not in cards | | Jun 2023 | Jun 2026 | ||
| mage-bench Models play Magic: The Gathering in a benchmark that compares strategic play across games. | mage-bench | Multi-agent games | Expert | Not in cards | GPT-5.4 Mini 1617.0 | Jan 2026 | Mar 2026 | ||
| Meeseeks Utility Rate Models follow complex instructions with feedback for up to 20 turns. The score measures fully usable responses as a percentage. | Meituan | Multi-turn conversation | Everyday | Not in cards | Claude Sonnet 4 90.5% | Apr 2025 | Aug 2025 | ||
| MemoryTrap Bench: PHP arrays, Go maps Models answer questions about memory use in PHP and Go. The grader compares answers with measurements, varying whether models see runtime versions or a measured fact. | Nazar Boyko | Speed and optimization | University | Not in cards | GPT-6.1 Sol 0.95 | Oct 2026 | Oct 2026 | ||
| MiJaBench Models respond to 43,961 English and Portuguese prompts across 16 minority groups. Judges measure how defense rates change with the target group and language. | Federal University of Goiás | Bias, fairness and politics | University | Not in cards | | Jan 2026 | Apr 2025 | ||
| MirageBench Models personalize six kinds of responses from three disclosed facts about each of 150 personas. A model judge classifies claims by evidential support. The primary over-inference rate counts stereotyped and fabricated claims. | Tencent LIGHTSPEED and Hong Kong University of Science and Technology | Faithful to sources | Everyday | Not in cards | | Aug 2026 | Apr 2026 | ||
| MissionReport Models write government status reports in Word from working files and a template, across 20 expert-written Department of Defense style assignments. Expert rubrics score each decision. | micro1 | Documents and spreadsheets | Expert | Not in cards | | Oct 2026 | Nov 2026 | ||
| MME-Reasoning Multimodal models answer logic questions grouped into inductive, deductive and abductive reasoning. | MME team (Alpha-Innovator) | Visual math and diagrams | University | Not in cards | | May 2025 | Jul 2025 | ||
| MTHv2 Models read Chinese Buddhist texts from scanned historical pages. The authors annotate characters, line transcriptions, reading order, and boundary lines. | South China University of Technology and Alibaba Group | Documents, charts and text in images | University | Not in cards | Kimi K2.6 86.4% | Jun 2020 | Jun 2026 | ||
| NoticIA Models summarize Spanish news articles in one sentence that answers the clickbait headline. The evaluator compares each summary with a human reference. | Iker García-Ferrero and colleagues | Faithful to sources | School | Not in cards | GPT-4 Turbo (Jan 2024) 32.0 | Apr 2024 | Feb 2024 | ||
| OmniMatBench Models answer questions and perform calculations across 19 materials science fields. The authors compare automated grades with human assessments. | Wanhao Liu and colleagues | Science questions | University | Not in cards | Claude Opus 4.7 0.37 | May 2026 | Apr 2026 | ||
| OR-License Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem. | ALLE AI | Tasks from real jobs | Expert | Not in cards | Grok 4.6 88.4% | Oct 2026 | Sep 2026 | ||
| OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. | DiscoAILab | Desktop apps | Expert | Not in cards | Claude Fable 5.1 73.7% | Sep 2026 | Sep 2026 | ||
| PARADE Audio Models answer questions about occupations and social status using audio clips. The evaluation tests whether voice cues bias their answers. | UCSC VLAA and the AHELM team | Bias, fairness and politics | Everyday | Not in cards | | Mar 2025 | May 2025 | ||
| QuantSightBench Models give 90% prediction intervals for 1,000 numerical forecasting questions with known outcomes. The main setting allows retrieval from a fixed news corpus. Mean Log Interval Score penalizes wide intervals and missed outcomes, while coverage measures how often intervals contain the answer. | ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, and Tübingen AI Center | Real-world events | Expert | Not in cards | GPT-5.1 6.70 | Apr 2026 | Mar 2026 | ||
| Questflow Finance Benchmark Agents query financial filings and market data with a fixed set of tools. They calculate metrics, rank securities, and submit answers that software checks against frozen references. | Questflow | Finance | University | Not in cards | GPT-6.1 Terra 100% | Oct 2026 | Oct 2026 | ||
| Realtime-Bench Agents act in 50 live game and web tasks where a late action fails: 30 web tasks (travel, shopping, services and security) and 20 games. The environment keeps moving while the model thinks. Three clocks separate speed from skill: Wall Clock runs on real time, Paused Clock stops while the model thinks, and Token Clock advances with each generated token. Scores are task success rates over three runs. | University of Texas at Austin, Ohio State University, UIUC, NVIDIA, Northwestern University and Steadyworks | Web browsing | Everyday | Not in cards | GPT-6 Astra 42.7% | Oct 2026 | Nov 2026 | ||
| Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. | Vals AI | Simulated robots | Expert | Not in cards | GPT-6.2 Astra 32.7% | Oct 2026 | Nov 2026 | ||
| RPEval Models respond in character to 9,018 scenarios. The test checks emotions, decisions, moral choices, and whether characters reveal knowledge outside their setting. | Yassine El Boudouri and colleagues | Overall chat quality | Everyday | Not in cards | | May 2025 | Jun 2025 | ||
| RuleTrek Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements. | RuleTrek authors | Language reasoning | Expert | Not in cards | GPT-6.1 Sol 55.3% | Oct 2026 | Sep 2026 | ||
| RWE-bench Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement. | Xplore AI | CAD models | Expert | Not in cards | Claude Fable 5.2 62% | Oct 2026 | Nov 2026 | ||
| SafePyramid Models identify rules violated by a conversation under a supplied policy. Rule matching rate averages four thresholds for false positive and false negative counts. | ByteDance and University of Melbourne | Refusing harmful requests | University | Not in cards | GPT-5.5 54.2% | Jun 2026 | May 2026 | ||
| SEC filing claim verification (400 items) Models classify whether an excerpt from a corporate filing supports a claim, contradicts it, or provides insufficient evidence. | Vals AI | Faithful to sources | University | Not in cards | Claude Sonnet 5.5 99.5% | Oct 2026 | Sep 2026 | ||
| SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. | Tsinghua University | Spatial reasoning and counting | Everyday | Not in cards | GPT-5 14.4% | Jun 2026 | Mar 2026 | ||
| SpatialWorld Digital Agents complete entertainment tasks in digital environments. The score is the published digital task success rate. | Tsinghua University | Spatial reasoning and counting | Everyday | Not in cards | | Jun 2026 | Mar 2026 | ||
| SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. | Tsinghua University | Spatial reasoning and counting | Everyday | Not in cards | GPT-5 14.4% | Jun 2026 | Mar 2026 | ||
| Speko Speech-to-Speech Benchmark (six concierge scenarios) Voice agents book flights, move reservations, and revise budgets during scripted calls. Speko checks whether agents wait for confirmation and use the latest details. | Speko | Customer service | Everyday | Not in cards | GPT-Live-1 0.91 | Oct 2026 | |||
| StudentBench AI tutors teach real students GRE material and are scored by the students measured learning gains. A second study has expert tutors compare AI lesson plans and practice problems. | Handshake AI Research | Other | University | Not in cards | Claude Opus 4.8 16.5 | Sep 2026 | Aug 2026 | ||
| SuperCLUE-Fin Models answer Chinese finance questions across 25 tasks in six categories. The evaluation covers financial knowledge and its application in practice. | SuperCLUE team | Finance | University | Not in cards | Kimi K3 88.7 | Apr 2024 | Jul 2026 | ||
| SusVibes Coding agents rebuild a feature in a real open-source Python project from a plain-language request. Each task comes from a historical vulnerability fix, so the original human version was insecure. Hidden functional tests and security tests check the result. Scores report the share of solutions that work and the share that are also secure. | Carnegie Mellon University (Lei Li's group) | Secure code and review | Expert | Not in cards | GPT-5.5 43.5% | Dec 2025 | Jun 2026 | ||
| SWE-bench Science Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix. | Fudan University OpenMOSS (Zhipeng Xu et al.) | Scientific code | Expert | Not in cards | GPT-6 Astra 50.4% | Aug 2026 | Sep 2026 | ||
| Taiwanese Mandarin Language Understanding Models answer questions from Taiwanese exams in traditional Chinese. The subjects range from school topics to professional qualifications. | Po-Heng Chen and colleagues | Knowledge in many languages | University | Not in cards | Claude 3 Opus 73.6% | Mar 2024 | Apr 2024 | ||
| TERMS-Bench Agents negotiate the price of a single item with simulated counterparts over up to 10 rounds of alternating offers. Counterparts follow six behavior families across three regimes. A program scores each agent from its logged actions on four separate axes: surplus captured, agreement calibration, belief error about the opponent and procedural violations. Extra modes score merchant profit and a bankroll over many sessions. | Stanford University | Multi-agent games | Everyday | Not in cards | Claude Opus 4.6 0.69 | May 2026 | Apr 2026 | ||
| TireSidewall-Bench: Multimodal Zero-Shot Tire OCR Models read the size markings on tire sidewalls from photographs. The grader compares extracted specifications with reference annotations. | Inusha Thathsara | Documents, charts and text in images | Everyday | Not in cards | | Oct 2026 | Oct 2026 | ||
| TutorCaseBench Tutors respond to students whose behavior interferes with learning. Case-specific rubrics grade the responses, and a separate track tests selection among possible interventions. | Seoul National University and collaborators | Other professions | University | Not in cards | Claude Opus 4.7 59.4% | Oct 2026 | Apr 2026 | ||
| UniGUI-Bench Agents interpret and operate graphical interfaces. The benchmark tests 11 capabilities and includes separate system evaluations. | AGI-Eval (Meituan) | Finding things on screen | Everyday | Not in cards | Kimi K2.5 80.9% | May 2026 | Apr 2026 | ||
| VibePrevisBench Coding agents write Blender Python that turns a script, photos, an opening frame or reference footage into a graybox 4D previs scene and render it. Rule checks score the exported scene against hidden facts, and a VLM judge checks that those facts are visible in the render. | KlingAI Research (Kling-Keye) | CAD models | Expert | Not in cards | GPT-6 Astra 77.7 | Nov 2026 | Sep 2026 | ||
| VISTA (Visually Impaired Scene and Task Annotations) Models describe scenes, answer questions, and give guidance to blind or low-vision users from glasses recordings. A judge scores responses from 0 to 5. | Harvard AI and Robotics Lab and collaborators | Video understanding | Everyday | Not in cards | | Oct 2026 | Mar 2026 | ||
| VoxCeleb2 Models compare two interview recordings and decide whether the same person speaks in both. | Joon Son Chung and colleagues, University of Oxford | Audio understanding and voice | Everyday | Not in cards | | Jun 2018 | May 2025 | ||
| 3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. | Prana Labs and University of California, Berkeley | Physical prediction | Expert | Not in cards | GPT-6 Astra 59.5 | Nov 2026 | Oct 2026 | ||
| AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. | University of Illinois Urbana-Champaign | General assistants | Everyday | Not in cards | GPT-5 67.8% | Jun 2026 | Apr 2026 | ||
| AgentCallBench: Tool-Calling Reliability Models operate a simulated store using tools. The grader checks tool choice, arguments, failure recovery, unnecessary calls, and refusals of impossible orders. | Abhisek Roy | Function calling | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| AgentCLUE Agents execute tasks in Chinese. | SuperCLUE | Apps and MCP servers | University | Not in cards | Claude 3.5 Sonnet (October 2024) 46.4 | Feb 2025 | |||
| Agentic Trader Agents search for market information and place simulated foreign exchange orders using live prices. Each agent manages its own positions, cash, and margin. | HKU Business School Artificial Intelligence Evaluation Lab | Finance | University | Not in cards | | Jul 2026 | Jul 2026 | ||
| AHELM Audio-language models are scored across 10 aspects with standard prompts. The suite combines existing datasets with two new ones, PARADE for stereotypes and CoRe-Bench for conversational reasoning. | Stanford CRFM | Audio understanding and voice | Everyday | Not in cards | | Aug 2025 | Sep 2025 | ||
| ASPI Agents complete simulated tasks with incomplete instructions. The authors compare adherence to user intent across matched versions of each task. | Scale AI | Deception and alignment | Everyday | Not in cards | | May 2026 | Apr 2026 | ||
| AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. | AutoLab team, with authors from UW, UCSD, Princeton, MIT, Google, NVIDIA, IBM and others | Machine learning engineering | Expert | Not in cards | Claude Opus 4.6 0.68 | Jun 2026 | Apr 2026 | ||
| BizBench Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success. | Ant International | Running a business | Expert | Not in cards | | Oct 2026 | Sep 2026 | ||
| BotChat Two instances of a model sustain a conversation. Judges assess the resulting dialogue. | OpenCompass and collaborators | Multi-turn conversation | Everyday | Not in cards | GPT-4 73.2 | Oct 2023 | Sep 2023 | ||
| CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. | NVIDIA + UC Berkeley + CMU + Stanford | Simulated robots | University | Not in cards | | Mar 2026 | Dec 2025 | ||
| Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. | Apart Research and collaborators | Capture-the-flag challenges | Expert | Not in cards | | Oct 2024 | Sep 2024 | ||
| CBT-Bench Level II Models identify cognitive distortions and classify patients’ core beliefs at two levels of detail. The evaluator compares the selected labels with the reference labels. | Mian Zhang and colleagues | Medicine | Expert | Not in cards | Claude Opus 5.5 0.63 | Oct 2024 | Sep 2026 | ||
| CFDCodeBench V2 Models write Python solvers for 24 fluid dynamics problems. The revised tasks specify output arrays and grid conventions and change required fields for some flows. | Rensselaer Polytechnic Institute and collaborators | Scientific code | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| CFDCodeBench v1 Replaced in Oct 2026 | Not in cards | Claude 3.5 Haiku 14.3% | May 2025 | Sep 2025 | |||||
| CFDQuery Models answer 90 multiple-choice questions about fluid simulation, including numerical stability, grid spacing, and discretization errors. The score is answer accuracy. | Rensselaer Polytechnic Institute and collaborators | Science questions | University | Not in cards | o3-mini 91.1% | May 2025 | Sep 2026 | ||
| CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. | Center for AI Safety | Deception and alignment | Expert | Not in cards | | Sep 2026 | Oct 2026 | ||
| CNFinBench Models answer financial questions and carry out financial workflows. The benchmark includes tasks on expertise, autonomy, and integrity. | OpenCompass and collaborators | Finance | Expert | Not in cards | | Dec 2025 | Sep 2026 | ||
| Co-Design Bench Agents revise existing applications from multimodal user requests. A judge checks browser and repository evidence against 2,331 expert criteria across 273 tasks. | Xilin Wei and collaborators | Web and app front ends | Expert | Not in cards | GPT-6 Astra 69.9% | Oct 2026 | Sep 2026 | ||
| CoCoNot Models get 1,000 requests they should not fully comply with, such as incomplete, unsupported or humanizing requests. A judge model rates whether each response complies. | Allen Institute for AI | Deception and alignment | Everyday | Not in cards | Claude 3 Sonnet 5.34% | Jul 2024 | Apr 2024 | ||
| Cost per Successful Task Agents solve 40 Terminal-Bench tasks. The benchmark divides spending across all attempts by the number of successful runs. | Arize AI and Fireworks AI | Terminal tasks | Expert | Not in cards | | Jul 2026 | Jul 2026 | ||
| CounselBench-Adv Models answer 120 questions written by mental health professionals. Each question targets one of six problems identified in earlier counseling responses. | University of Southern California | Wellbeing and sycophancy | Expert | Not in cards | GPT-4o 0.37 | May 2025 | Sep 2026 | ||
| CounselBench-Eval Models answer 100 questions that people posted for therapists. Mental health professionals rated responses on six dimensions. | University of Southern California | Medicine | Expert | Not in cards | | May 2025 | Sep 2026 | ||
| CTI-REALM Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry. | CTI-REALM authors, implemented in UK AISI inspect_evals | Finding and exploiting real bugs | Expert | Not in cards | Claude Opus 4.6 63.7% | Mar 2026 | Feb 2026 | ||
| Cua-Bench EDA (KiCad v1) Agents build or edit KiCad schematics through the desktop interface. The grader checks the resulting netlist and reports full passes, partial passes, and mean reward. | Snorkel AI and Cua AI | Electronics | Expert | Not in cards | GPT-5.5 24% | Jun 2026 | Jul 2026 | ||
| EduArt Models answer art history questions from Italian secondary schools and US Advanced Placement exams. The authors compare 12 models across seven question formats. | University of Bologna and collaborators | Exam questions | School | Not in cards | | Jul 2026 | May 2026 | ||
| Embodied First Exam Coding agents answer frozen robot tasks from many benchmark suites with a program. Each program is verified in a separate sandbox, run on Harbor. | Northwestern University and collaborators (CMU, UC Berkeley, Stanford and others) | Simulated robots | Everyday | Not in cards | Claude Fable 5.2 49.6% | Oct 2026 | Nov 2026 | ||
| EnviroExam Models answer multiple choice questions from 42 environmental science courses. The composite index discounts mean accuracy when performance varies across courses. | Harbin Institute of Technology | Science questions | University | Not in cards | | May 2024 | Feb 2024 | ||
| ExpertLongBench Models write documents for 11 professional tasks. A judge compares facts in each document with a reference checklist and scores precision and recall. | University of Michigan and collaborators | Tasks from real jobs | Expert | Not in cards | | Jun 2025 | Aug 2025 | ||
| ExplorationBench Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help. | Tencent Hunyuan | Research science | Research frontier | Not in cards | Claude Opus 5 89% | Sep 2026 | Sep 2026 | ||
| Flawed Fictions Models detect plot holes in stories and identify the line responsible. The hub port uses an 83-example holdout by default. | Kabir Ahuja and collaborators | Reading comprehension | University | Not in cards | Claude 3.7 Sonnet 0.66 | Apr 2025 | Feb 2025 | ||
| FoamBench V2 Agents build and run 110 Basic and 16 Advanced OpenFOAM cases. Revised instructions specify geometry, meshes, and boundary conditions for the same named cases. | Rensselaer Polytechnic Institute and collaborators | Apps and MCP servers | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| FoamBench v1 Replaced in Oct 2026 | Not in cards | o3-mini 22.5% | May 2025 | Sep 2025 | |||||
| Fund Smart Routing Agent Models choose a fund for a transaction. Ramp measures how often a user overrides that choice. | Ramp Labs | Finance | University | Not in cards | GPT-5.1 100% | Mar 2026 | Mar 2026 | ||
| Genspark Slides Benchmark Models build finished slide decks from 200 real production tasks in the Genspark Slides agent harness. An internal grader scores quality, completion, content and design. | Genspark | Documents and spreadsheets | Everyday | Not in cards | Claude Fable 5.1 83.6% | Sep 2026 | |||
| GitTaskBench Code agents solve 54 real tasks across 7 modalities and 7 domains by using a given GitHub repository, from understanding and setup to running it. | QuantaAlpha | Fixing real repositories | Expert | Not in cards | Claude 3.7 Sonnet 48.1% | Aug 2025 | Apr 2025 | ||
| Hebbia Financial Services Benchmark Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion. | Hebbia | Finance | Expert | Not in cards | Claude Opus 5.5 74.6% | Oct 2026 | Oct 2026 | ||
| Investor's Last Exam Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria. | Prophet Lab | Finance | Expert | Not in cards | GPT-6 Astra 68.1 | Oct 2026 | Sep 2026 | ||
| ISA-Bench Models write programs for five games with restricted instruction sets. A verifier checks each program and gives feedback before the next attempt. | IIT Hyderabad and Microsoft Research India | Puzzles and chess | University | Not in cards | | Sep 2026 | Apr 2026 | ||
| KARLBench Agents answer questions using vector search over fixed document collections. The suite measures six kinds of evidence gathering and reasoning. | Databricks AI Research | Questions that need search | University | Not in cards | Claude Opus 4.6 67.5 | Mar 2026 | Feb 2026 | ||
| LLM-AggreFact Models judge whether claims are supported by a grounding document across 11 pooled datasets, such as RAGTruth and ExpertQA. Scored by balanced accuracy. | UT Austin + Salesforce AI Research (Tang, Laban and Durrett) | Faithful to sources | Everyday | Not in cards | GPT-4o 75.9% | Apr 2024 | Dec 2024 | ||
| Local LLM Arkanoid Test Agents build a mouse-controlled brick breaker in one HTML file. They can revise it before a separate Claude agent checks 11 requirements in a browser. | AOX.cz | Web and app front ends | University | Not in cards | gpt-oss-20b 11.0 | May 2026 | Apr 2026 | ||
| Local LLM Tetris Test Agents build Tetris in one HTML file and can revise it after playing. A separate Claude agent checks piece movement, line clearing, scoring, and restart. | AOX.cz | Web and app front ends | University | Not in cards | | May 2026 | Apr 2026 | ||
| MCP-SafetyBench Agents complete tool-use tasks across five domains. The benchmark reports both task completion and resistance to adversarial inputs. | MCP-SafetyBench authors | Jailbreaks and prompt injection | University | Not in cards | o4-mini 21.2% | Dec 2025 | Sep 2025 | ||
| Medical Chronology LLM Benchmark Models extract medical timelines from six synthetic document sets in three independent rounds. Checks compare extracted events with reference timelines and assess semantic fidelity, formatting, date order, and hallucinations. A weighted composite combines these checks with ROUGE and extraction F1. | Superinsight, Inc. | Medicine | Expert | Not in cards | Claude Opus 4.6 88.9 | Apr 2026 | Mar 2026 | ||
| MedSafetyBench Models respond to requests that conflict with medical ethics. A judge rates harmfulness on a five-point scale using a held-out set of 900 requests. | Tessa Han and colleagues | Refusing harmful requests | Everyday | Not in cards | | Mar 2024 | Jul 2024 | ||
| micro1 Medical Index Models are scored on the equal-weight average of micro1 medical benchmarks, currently RealmPathology, PharmaBench and WellnessBench. | micro1 | Medicine | Expert | Not in cards | GPT-6 Astra 71% | Oct 2026 | Nov 2026 | ||
| MME-CoT Models reason step by step on image questions in math, science, OCR, logic, space-time and general scenes. A judge scores reasoning quality, robustness and efficiency. | MME team (CUHK MMLab and others) | General visual questions | University | Not in cards | Kimi k1.5 64.2% | Feb 2025 | Jan 2025 | ||
| MMEnterprise-Bench Enterprise agents answer 1,613 questions over workplace documents, chats and figures that change over time. They must resolve conflicting versions and say when evidence is not enough. | MLV Lab (mlvlab) | Simulated companies | Expert | Not in cards | GPT-5.6 Sol 68% | Oct 2026 | Sep 2026 | ||
| MMK12 Models use images to answer school mathematics and science questions. The test split contains 2,000 problems with reference answers. | MM-EUREKA team | Visual math and diagrams | School | Not in cards | o1 73.9% | Apr 2025 | Mar 2025 | ||
| MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. | University of Tokyo (Li Lab) and collaborators | Knowledge in many languages | University | Not in cards | | May 2025 | Apr 2025 | ||
| MMTU Models answer about 30,000 questions over real tables across 25 tasks, such as data cleaning, table joins, column and table transformation, table question answering and turning language into code. The tasks come from decades of computer science research on tabular data. Answers are checked automatically against references. | Microsoft and University of Michigan | Data science and SQL | Expert | Not in cards | GPT-5 69.6% | Jun 2025 | Aug 2025 | ||
| NeurIPS 2025 Experts Benchmark Models answer tasks submitted by participants at NeurIPS 2025 in San Diego. Kaggle reports an overall score and individual task outcomes. | Kaggle | Language reasoning | University | Not in cards | | Dec 2025 | Dec 2025 | ||
| NOTA Models extract facts from sheet music, convert score images to ABC notation, and analyze musical structure. The authors report extraction rates and separate transcription distances. | Mingni Tang and colleagues | Documents, charts and text in images | University | Not in cards | | Jun 2024 | May 2024 | ||
| OdysseyArena Agents explore interactive environments built from four transition primitives and must work out the hidden dynamics over long horizons. OdysseyArena-Lite has 120 standard tasks. | OdysseyArena authors | Puzzles and chess | Expert | Not in cards | | Feb 2026 | Dec 2025 | ||
| OmniGameArena Models play games while response latency advances the game clock. | OmniGameArena authors | Video games | Everyday | Not in cards | GPT-5.5 0.47 | Jun 2026 | Apr 2026 | ||
| OmniRouting Models route copper traces between pads on printed circuit boards. The evaluator counts required nets that connect without design rule violations. | Pennsylvania State University and collaborators | Electronics | University | Not in cards | GPT-5.5 28% | Jul 2026 | May 2026 | ||
| OpenGameEval Debug Evals Agents diagnose and fix faults in Roblox Studio games. The authors created 30 debugging tasks from 15 scenarios. | Roblox | Fixing real repositories | University | Not in cards | Claude Fable 5 64.7% | Mar 2026 | Jun 2026 | ||
| PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. | UniPat AI | Research science | Research frontier | Not in cards | GPT-6 Astra 14% | Oct 2026 | Sep 2026 | ||
| PeerQA Models answer peer reviewers' questions about a scientific article using evidence from the article. | UKP Lab | Research science | Expert | Not in cards | | Feb 2025 | May 2025 | ||
| PMBench Agents search product managers’ notes to identify evidence across customer conversations and answer questions about product requirements. | Databricks AI Research | Questions that need search | University | Not in cards | Claude Opus 4.6 46.1 | Mar 2026 | Feb 2026 | ||
| Policy Agent Models approve expenses, reject them, or request human review. Ramp compares their decisions with human expense reviews. | Ramp Labs | Finance | University | Not in cards | GPT-5.4 100% | Mar 2026 | Mar 2026 | ||
| Polish Medical Exams Models answer Polish medical, dental, and specialist examination questions. The benchmark also includes English editions of the medical and dental licensing exams. | Adam Mickiewicz University and collaborators | Medicine | Expert | Not in cards | GPT-4o 79.9% | Nov 2024 | Sep 2024 | ||
| ProcessBench Models read step-by-step solutions to GSM8K, MATH, OlympiadBench and Omni-MATH problems and name the earliest wrong step, or say none is wrong. Scored by F1. | Qwen Team, Alibaba Group | Competition problems with answers | Competition | Not in cards | o1-mini 87.9% | Dec 2024 | Dec 2024 | ||
| Progress-Bench Vision-language models estimate how far a task has progressed from partial observations, such as frames of a robot or human doing a job. The benchmark varies how the demonstration is shown and the camera viewpoint, and it includes cases with no valid answer. | ProgressLM team | Spatial reasoning and counting | Everyday | Not in cards | GPT-5 mini 20.9 | Jan 2026 | Oct 2025 | ||
| ProofRank Models write proofs for hard competition problems. An LLM judge compares correct proofs pairwise on conciseness, computational ease, simplicity, diversity and adaptivity, giving Elo ratings. | MathArena (ETH Zurich SRI Lab and INSAIT) | Written proofs | Competition | Not in cards | GPT-5.4 1,429 | May 2026 | Mar 2026 | ||
| Q-Bench Multimodal models are tested on low-level vision: perceiving image distortions, describing them and predicting quality scores. | Nanyang Technological University and collaborators | General visual questions | Everyday | Not in cards | Qwen-VL-Max 73.9% | Sep 2023 | Jan 2024 | ||
| RealWorkDecision Agents review a frozen company workspace and predict the decision an employee made. Expert-written rubrics allow valid alternatives. | micro1 | Tasks from real jobs | University | Not in cards | Grok 4.7 43.8% | Oct 2026 | Oct 2026 | ||
| RedstoneBench Agents design redstone circuits in Minecraft. A game server tests whether each circuit works and measures the space it occupies. | Adam Holter | Video games | Expert | Not in cards | GPT-6.2 Astra 88.7 | Nov 2026 | Nov 2026 | ||
| Right Answer, Wrong Reason: ML Judgment Calls Models make 36 decisions about tabular machine learning. The benchmark compares choosing from options with giving advice freely against measured outcomes. | starkhushi | Machine learning engineering | University | Not in cards | Claude Sonnet 5.5 32.0 | Oct 2026 | Sep 2026 | ||
| RoCS-MT v2 Models translate Reddit posts with revised spelling annotations and corrected references. The authors combine COMETKiwi and MetricX rankings for translations of the original text. | Inria | Translation | Everyday | Not in cards | GPT-4.1 7.00 | Nov 2025 | Apr 2025 | ||
| RoCS-MT v1 Replaced in Nov 2025 | Not in cards | Oct 2023 | |||||||
| RSLiveBench-UAV Models answer questions about vehicles in drone images. The benchmark checks perception and spatial reasoning across different views and adds newly captured scenes. | East China Normal University and collaborators | Embodied reasoning | Everyday | Not in cards | GPT-6 Astra 59.9 | Nov 2026 | Oct 2026 | ||
| RTLScout Bench (suite-v2) Agents rewrite five circuits to reduce the product of chip area and signal delay. The leaderboard ranks configurations by their mean rank across circuits. | Felix Arnold | Electronics | Expert | Not in cards | MiMo-V2.6 Pro 1.00 | Oct 2026 | Oct 2026 | ||
| scBench-Long (original study) Agents complete 21 tasks in long single-cell data analysis workflows. The authors compare their answers with results from the original analyses. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 4.8 25.4% | Jun 2026 | May 2026 | ||
| SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. | Tianyun Zhong and colleagues | Research science | Expert | Not in cards | GPT-6 Astra 80.9 | Oct 2026 | Sep 2026 | ||
| SIM-VAIL Chatbots respond to simulated users with psychiatric vulnerabilities. A model judge scores behaviors across the conversation, including responses that reinforce a user’s vulnerability. | Veith Weilnhammer and colleagues | Wellbeing and sycophancy | Expert | Not in cards | GPT-6 Astra 1 | Feb 2026 | Sep 2026 | ||
| SLATE Models generate slides that teach unfamiliar language patterns. Simulated learners take tests before and after the lesson and apply the rules to new examples. | Beijing Normal University and collaborators | Other professions | University | Not in cards | Claude Opus 4.7 5.46% | Sep 2026 | Apr 2026 | ||
| Spider 2.0-AIFunc Agents generate SQL that uses AI functions to solve 465 tasks across 125 databases. The evaluator compares execution results with reference results. | UC San Diego, Snowflake AI Research and University of Hong Kong | Data science and SQL | University | Not in cards | Claude Opus 4.6 70.3% | Jul 2026 | Mar 2026 | ||
| STALE Agents with long-term memory must notice when a later observation quietly makes an earlier memory false. The benchmark has 400 expert-validated conflict scenarios and 1,200 queries over more than 100 everyday topics, with contexts up to 150K tokens. Queries test whether the model spots the outdated belief, rejects a false premise and acts on the new state. | HKUST NLP Group | Learning from the context | Everyday | Not in cards | | May 2026 | Mar 2026 | ||
| SuperCLUE-Image Models generate images from Chinese prompts. The benchmark scores text rendering, visual quality, prompt adherence and scene composition. | SuperCLUE | Image generation and editing | Everyday | Not in cards | GPT-Image-2.5 Sunburst 94.3 | Jun 2024 | |||
| SuperCLUE-Law Models respond to Chinese legal questions. | SuperCLUE | Law | Expert | Not in cards | Kimi K3 92.1 | Aug 2026 | |||
| SuperGLUE Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score. | NYU, Facebook AI, University of Washington, and DeepMind | Language reasoning | Everyday | Not in cards | | May 2019 | Apr 2022 | ||
| SWE-CoWork Coding agents join a simulated engineering team on 101 real repositories and must gather scattered requirements from simulated colleagues and workplace records before implementing them. Hidden tests score the share of new-behavior tests passed and whether the whole task passes. | TokenWave AI | Fixing real repositories | Expert | Not in cards | GPT-6 Astra 7.92% | Nov 2026 | Sep 2026 | ||
| SWE-Serve Coding agents solve 53 production inference-engineering tasks derived from merged SGLang pull requests. Hidden tests measure task success, cost, agent steps and wall time. | NVIDIA | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 75.5% | Oct 2026 | Jul 2026 | ||
| The Aggregate Asteroid Numbers Models give the permanent numbers of 48 named asteroids from memory. The scorer checks each answer against the NASA Jet Propulsion Laboratory catalog. | Mikhail Doroshenko (The Aggregate) | Trivia and open questions | Expert | Not in cards | | Sep 2026 | Nov 2026 | ||
| TREC BioGen 2024 Systems retrieve biomedical literature and synthesize evidence into reports that answer a question. | TREC BioGen track organizers | Medicine | Expert | Not in cards | Claude Opus 4.6 79.9 | Feb 2026 | |||
| VCBench Models predict venture outcomes from founder profiles. The benchmark emphasizes precision through the F0.5 score. The hub adaptation substitutes synthetic profiles for the original data. | University of Oxford and Vela Research | Finance | Expert | Not in cards | GPT-6 Sol 33.7% | Sep 2025 | Sep 2026 | ||
| WeatherQA (severe weather) Models read weather maps to identify a threatened area and classify its severe weather potential. The second score includes only cases with the correct area. | University of Washington and collaborators | Science questions | Expert | Not in cards | | Jun 2024 | Dec 2025 | ||
| WeaveBench Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts. | Microsoft and collaborators | Desktop apps | Expert | Not in cards | Claude Opus 4.7 41.2% | Jun 2026 | Jun 2026 | ||
| Web Clone Bench Agent gets a running reference app (Gmail, Jira, Figma, etc.) and 8 hours to build a clone. Expert-written workflows are replayed as end-to-end tests. | Plato | Building whole projects | Research frontier | Not in cards | Claude Fable 5.2 62% | Oct 2026 | Nov 2026 | ||
| AgencyBench Agents complete 138 long tasks drawn from daily AI use, grouped into 32 scenarios that cover six agentic capabilities such as coding, front-end work, game development and research. A typical task takes about 90 tool calls, 1 million tokens and hours of work. A simulated user gives feedback during the run, and a Docker sandbox checks the deliverables against task rubrics. | Shanghai Innovation Institute (SII) GAIR Lab | Long terminal sessions | Expert | Not in cards | GPT-5.2 56.5% | Jan 2026 | Dec 2025 | ||
| AgentCLUE-Mobile (offline) Agents select actions in an offline mobile interface. | SuperCLUE | Phones | Everyday | Not in cards | GLM-4.5V 88.4 | Aug 2025 | |||
| AgenticFinBench Agents complete 230 financial requests in two languages. Graders apply task-specific rubrics and audit unsupported claims against external evidence. | NashNova Research | Finance | Expert | Not in cards | GPT-5.6 Sol 89.2 | Nov 2026 | Sep 2026 | ||
| AgroBench Models answer illustrated questions from agronomists about crops, diseases, pests, weeds, machinery and farming practices. The overall score averages accuracy across seven tasks. | Risa Shinoda and colleagues | General visual questions | University | Not in cards | GPT-4o 73.5% | Jul 2025 | Aug 2024 | ||
| AIRTBench Models act as autonomous AI red teamers on 70 black-box capture-the-flag challenges from Dreadnode's Crucible platform. Each challenge hides a flag behind an AI or ML system, and the model writes Python code to attack it with methods such as prompt injection, evasion and model inversion. A challenge counts as solved when the model submits the correct flag. | Dreadnode | Capture-the-flag challenges | Expert | Not in cards | Claude 3.7 Sonnet 46.9% | Jun 2025 | May 2025 | ||
| ANCHOR Models explain answers to audiovisual questions and locate the supporting evidence. The benchmark includes time intervals, image regions, and relevant sounds. | Joel Julin and colleagues | Audio understanding and voice | Everyday | Not in cards | | Oct 2026 | Apr 2026 | ||
| AsyncTool Agents coordinate several tool-use tasks while tool responses arrive after simulated delays. The evaluator checks function calls, parameters, task completion, and the final environment state. Scheduling metrics measure whether agents use waiting time effectively. | University of Science and Technology of China and University of Toronto | Function calling | Everyday | Not in cards | GPT-4.1 38.1% | May 2026 | Sep 2025 | ||
| AutoBencher Models answer generated benchmark questions. The framework searches for topics and produces tests of knowledge, multilingual reasoning, and math. | Xiang Lisa Li and colleagues | Language reasoning | University | Not in cards | Claude 3 Opus 55% | Jul 2024 | Apr 2024 | ||
| AutoformBench Agents fill in missing Lean proofs for mathematical statements and submit their solutions to a clean grading workspace. | Meta FAIR | Formal proofs | Expert | Not in cards | GPT-6 Astra 82.4% | Oct 2026 | Sep 2026 | ||
| Automotive-ENV Agents operate climate controls in an Android Automotive OS simulator. A validator checks the screen and vehicle signals after requests such as turning on the defroster. | University of Technology Sydney and University of Liverpool | Apps and MCP servers | Everyday | Not in cards | Kimi K2.5 49.3% | Sep 2025 | Feb 2026 | ||
| BallotBench Nine frontier agents forecast 863 contests in the 2026 US midterms each day until polls close. Proper scoring rules grade forecasts against certified results, beside Kalshi, Polymarket and expert ratings. | Thoughtful Lab | Real-world events | Expert | Not in cards | | Oct 2026 | Oct 2026 | ||
| Big Code Models Leaderboard Models generate programs in multiple languages. The board compares pass rates and averages each model’s win rate across languages. | BigCode | Function-level problems | University | Not in cards | | Jul 2023 | Nov 2024 | ||
| BIRD-CRITIC Models fix real user SQL issues in MySQL, PostgreSQL, SQL Server and Oracle applications. Test cases check each fix. | BIRD team (HKU) | Data science and SQL | Expert | Not in cards | o1-preview 35.5% | Jan 2025 | Mar 2025 | ||
| BrokenMath Models are asked to prove false versions of 2025 competition problems. An LLM judge counts how often a model writes a proof instead of rejecting the statement. | INSAIT and ETH Zurich SRI Lab | Written proofs | Competition | Not in cards | GPT-5 29% | Oct 2025 | Sep 2025 | ||
| Browser Use LinkedIn Agents answer plain-language questions about LinkedIn using recorded page snapshots. The leaderboard reports correctness, cost, and time per task. | lowgular | Web browsing | Everyday | Not in cards | Claude Haiku 5.5 97.9 | Nov 2026 | Nov 2026 | ||
| Bug in the Code Stack Models find a syntax error inserted into a long Python codebase. Hamming measures retrieval accuracy across context lengths and error positions. | Hamming AI and Wat.ai | Finding facts in long inputs | School | Not in cards | GPT-4o 82.7% | Mar 2025 | |||
| ChemArgus Models solve chemistry competition and textbook problems with text and figures. A model judge awards partial credit for each subquestion. | Hanyao Guo and colleagues | Science questions | Competition | Not in cards | Claude Opus 5 64.9% | Oct 2026 | Sep 2026 | ||
| clembench Chat models play rule-based dialogue games in self-play, such as Taboo, Wordle and reference games. The clemscore combines how often a model follows the game rules at all with how well it plays when it does. Version 4.0 runs 14 games in 30 languages. | University of Potsdam (Foundations of Computational Linguistics) | Multi-agent games | Everyday | Not in cards | GPT-5.4 88.8 | May 2023 | Jul 2026 | ||
| ClinHallu Models answer medical image questions and produce intermediate reasoning. The evaluation reports answer accuracy and separate hallucination rates for visual recognition, knowledge recall, and reasoning integration against annotated reasoning traces. | Alibaba DAMO Academy | Medicine | Expert | Not in cards | | Jun 2026 | Feb 2026 | ||
| CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. | M-A-P and collaborators | General visual questions | University | Not in cards | | Jan 2024 | Aug 2024 | ||
| CodeElo Models solve Codeforces contest problems through the platform's judging system. The harness converts contest outcomes into human-comparable Elo ratings and percentiles. | Qwen team, Alibaba | Competitive programming | Competition | Not in cards | o1-mini 1,578 | Jan 2025 | Nov 2024 | ||
| ContextBench Coding agents resolve 1,136 issues from 66 repositories in eight programming languages, and the benchmark checks which code they look at along the way. Each task has human-annotated gold context, and the tracker scores context recall, precision and efficiency over the agent's trajectory. It adds these process measures to the usual issue-resolution result. | Nanjing University and University College London | Fixing real repositories | Expert | Not in cards | GPT-5 60.6% | Feb 2026 | May 2026 | ||
| CookBench v1 (CookWing) Models give ingredient quantities for carbonara, pastitsio, and melon pan. A run passes when all required amounts and ratios fall within the dish’s specified ranges. | Tatsuo, CookWing | Language reasoning | Everyday | Not in cards | GPT-5.6 Sol 40% | Aug 2026 | Jul 2026 | ||
| Creative Control: LLMs as a Film Crew Models plan a film from concept to shot list while a director revises the brief. Separate tasks evaluate continuity and camera choices. | ChenLuFried | Creative writing | University | Not in cards | GPT-5.5 0.94 | Oct 2026 | Aug 2026 | ||
| CRMArena Agents do nine customer-service CRM tasks for service agent, analyst and manager personas inside a real Salesforce org. The org holds 16 linked object types with seeded hidden patterns. | Salesforce AI Research | Customer service | Everyday | Not in cards | o1 64.3% | Nov 2024 | Jan 2025 | ||
| CRMArena-Pro Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private. | Salesforce AI Research | Customer service | Everyday | Not in cards | | May 2025 | Apr 2025 | ||
| CUA-bench Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out. | Vals AI | Video games | Everyday | Not in cards | GPT-6 Astra 19.2% | Sep 2026 | Oct 2026 | ||
| DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. | Robocurve | Real robots | Everyday | Not in cards | GPT-6 Astra 40% | Oct 2026 | Oct 2026 | ||
| DECIDE Models select decisions from short voice-call clips and conversation history across 4,867 questions, 1,081 clips, and 32 tasks. Human reference labels determine accuracy. The default board ranks by DECIDE Balance Score, which combines accuracy, cost, and latency on a zero-to-100 scale. | Tazo Labs and Bolna | Audio understanding and voice | Everyday | Not in cards | | Nov 2026 | Sep 2026 | ||
| DeepTerminalBench Agents get one attempt at each of 50 deep engineering tasks in Docker, in the Terminal-Bench 2.0 format. A task passes only if every hidden test passes. Tasks were calibrated so Claude Opus 4.6 still fails some at pass@4. | UniPat AI | Terminal tasks | Expert | Not in cards | Claude Opus 4.7 34% | May 2026 | Apr 2026 | ||
| DISBench Multimodal agents search raw visual histories over several steps to find target images from implicit context cues. | NLPIR Lab, Renmin University of China | General visual questions | Everyday | Not in cards | Claude Opus 4.5 28.7% | Feb 2026 | Dec 2025 | ||
| DuetBench Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head. | Decagon | Customer service | Expert | Not in cards | Claude Opus 5 1,035 | Jun 2026 | Aug 2026 | ||
| EnterpriseClawBench Agents do 852 tasks rebuilt from real proprietary workplace agent sessions, each with fixtures, hard rules and semantic rubrics. | Frontis AI | Documents and spreadsheets | Expert | Not in cards | GPT-5.5 66.3% | Jun 2026 | Jun 2026 | ||
| Era by Eon Generates a simulated enterprise (Salesforce, Slack, Jira, Zendesk, etc.) with exact ground truth. Agents answer questions through MCP/REST. | Eon | Apps and MCP servers | University | Not in cards | Claude Opus 4.8 76.8% | Sep 2026 | Jul 2026 | ||
| EusProficiency Models complete Basque language exercises from the official C1 proficiency examination. The questions test vocabulary and grammar. | Julen Etxaniz and colleagues | Reading in many languages | University | Not in cards | GPT-4 Turbo (Jan 2024) 56.7% | Mar 2024 | Jan 2024 | ||
| FEM-Bench Models implement finite element functions and write tests for them. The evaluator checks numerical outputs and whether the tests distinguish reference functions from faulty implementations. | Lejeune Lab at Boston University and collaborators | Scientific code | University | Not in cards | | Aug 2025 | Nov 2025 | ||
| Fiction Bench (Foreverse) Models continue two Chinese novels for 20 rounds and receive their earlier writing as context. Reviewers rank anonymous continuations for each genre. | Foreverse Research | Creative writing | University | Not in cards | | Jul 2026 | Jul 2026 | ||
| FinCrime Agent Benchmark Agents screen customers for sanctions, political exposure, and adverse media. A case passes when agents find the relevant alerts and resolve them under the bank's policy. | Castellum.AI | Finance | University | Not in cards | GPT-6 Astra 71.1% | Sep 2026 | Sep 2026 | ||
| FinHopper Agents research financial facts through up to six dependent searches. Intermediate answer keys show where an incorrect company, date, or indicator derails the search. | South China University of Technology and Tencent | Finance | University | Not in cards | Claude Sonnet 4.5 67.3% | Oct 2026 | Dec 2025 | ||
| Firecrawl Search Bench Agents answer hard research questions with web search. A deterministic grader checks each answer against reference facts. | Firecrawl | Deep research | Expert | Not in cards | Claude Opus 5.5 79% | Nov 2026 | Oct 2026 | ||
| FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. | AGI-Eval (Meituan) | Documents and spreadsheets | University | Not in cards | | Nov 2026 | Sep 2026 | ||
| FrontierRSI Agents do 27 open-ended AI research and engineering tasks across pre-training, post-training and harnesses, inference serving and GPU kernels. Each trial is scored against the best valid result any model reached. | Abundant AI | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 34.3% | Nov 2026 | Nov 2026 | ||
| GAIK Ajokortti transcription test Models transcribe one three-minute Finnish dental webinar excerpt with two speakers. The project reports word error rate and separately counts errors that change the meaning. | GAIK project, Haaga-Helia University of Applied Sciences | Speech recognition and translation | Everyday | Not in cards | | Jan 2026 | Jul 2026 | ||
| GameCraft-Bench Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together. | CUHK Shenzhen | Building whole projects | Expert | Not in cards | GPT-6 Astra 75.9% | Jun 2026 | Sep 2026 | ||
| GeoDrive-Bench Models choose driving actions from road images and local traffic conventions. The benchmark covers six countries and omits explicit country labels. | Yingzi Ma and colleagues | Embodied reasoning | Everyday | Not in cards | GPT-5.4 91.9% | Jun 2026 | Mar 2026 | ||
| German fruit-story antipresuppositions Models read short German stories and choose an article or quantifier that fits the number of fruits. A later task asks whether a continuation sounds appropriate. | Judith Sieker and Sina Zarrieß | Reasoning in many languages | Everyday | Not in cards | GPT-5 100% | Dec 2023 | Sep 2025 | ||
| GomokuBench Models place stones on a 19 by 19 board against a search engine. The score rewards wins, quicker wins, and longer losses. | Homer Quan | Multi-agent games | Everyday | Not in cards | | Apr 2026 | Oct 2026 | ||
| Grade Your Own Homework: Can LLMs Judge Each Other Models solve problems and grade answers, including their own. The benchmark varies answer order, author names, and stated confidence against labels computed by code. | Anish_23101 | Language reasoning | School | Not in cards | | Oct 2026 | Sep 2026 | ||
| Graduate Admissions Agent Benchmark Agents advise students on graduate admissions and revise plans when the student's information changes. Seventy Chinese-language tasks span 23 cases and 83 answer stages. Separate reviewers score responses against task-specific rubrics on a 100-point scale. | 屿智同行 | Other professions | Expert | Not in cards | GPT-6.1 Sol 79.1 | Oct 2026 | Sep 2026 | ||
| HalluDial Models detect hallucinations in dialogue. | BAAI FlagEval | Hallucination rates | Everyday | Not in cards | GPT-4 Turbo (Jan 2024) 80.2 | Jun 2024 | May 2024 | ||
| HarvestBench Agents drive simulated tractors and decide how much fuel to spend avoiding animals. The benchmark measures how often they hit animals as avoidance costs change. | Jasmine Brazilek and colleagues | Deception and alignment | Everyday | Not in cards | GPT-5.6 Terra 0.40% | Sep 2026 | Sep 2026 | ||
| IMProofBench Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically. | ETH Zurich (Johannes Schmitt et al.) with a contributor community | Research mathematics | Research frontier | Not in cards | GPT-5.4 49.3% | Sep 2025 | Mar 2026 | ||
| Indic-KCC-Agri-Advisory-Benchmark Models answer crop questions from India’s Kisan Call Center in English and ten Indian languages. A judge compares each answer with a human advisor’s response. | Sthānika AI | Knowledge in many languages | University | Not in cards | | Sep 2026 | Apr 2026 | ||
| INQUIRE-Rerank Models reorder 100 photographs supplied for each ecological query. The benchmark measures average precision among the first 50 results. | INQUIRE team | General visual questions | University | Not in cards | GPT-4o 62.4% | Jun 2024 | Mar 2025 | ||
| INQUIRE-Fullrank Replaced in Jun 2024 | Not in cards | Jun 2024 | |||||||
| Into the Unknown Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check. | Prometheus-Eval team with US DOE national laboratories (Seungone Kim et al.) | Research science | Research frontier | Not in cards | Claude Opus 4.8 0% | Oct 2026 | Sep 2026 | ||
| KernelBench (kernelbench.com) Coding agents write GPU kernels for real model workloads, such as fused MoE layers, paged attention and FP8 matrix multiplies. The site groups tasks into suites: Mega (one decode task), CUDA (4 tasks) and Hard (6 tasks). A run counts only if it passes an audit of correctness. Scores are speedup over a baseline or percent of the hardware roofline. | Elliot Arledge (independent) | GPU kernels | Expert | Not in cards | | Jul 2026 | Oct 2026 | ||
| L2-Bench Models design lessons, give learners feedback, and participate in language practice. A model judge grades each response against teaching criteria. | James Edgell and colleagues | Other professions | University | Not in cards | Claude Opus 4.7 85.5 | Jul 2026 | Apr 2026 | ||
| LIBRA (Russian Long Context) Models read long Russian texts and answer questions or generate responses. The suite groups tasks by difficulty and varies input length. | Igor Churin and colleagues | Reasoning over long documents | University | Not in cards | | Aug 2024 | Feb 2026 | ||
| LMentry Models perform simple language tasks such as identifying words with a given letter or choosing the longer word. Task-specific rules check the answers. | Avia Efrat and colleagues | Language reasoning | Everyday | Not in cards | davinci-002 66.1% | Nov 2022 | Mar 2022 | ||
| LongCoT Models solve long reasoning problems across five domains. Task-specific programs verify answers, with optional judge fallback for ambiguous mathematics and chemistry responses. | Sumeet Ramesh Motwani and colleagues | Reasoning over long documents | Expert | Not in cards | GPT-5.2 25.1% | Apr 2026 | Feb 2026 | ||
| Longtail-SWE Agents resolve 202 GitHub issues across 13 programming languages with limited benchmark coverage. Hidden tests check the fix and verify that previously passing behavior still works. | Vladimir Dobrovolskii and colleagues | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 81.5% | Nov 2026 | Sep 2026 | ||
| LoopsBench Agents complete 112 software tasks with dependencies among smaller development units. Executable tests measure resolved tasks and the fraction of units that pass. | Microsoft | Long terminal sessions | Expert | Not in cards | Claude Opus 4.7 25% | Jul 2026 | Apr 2026 | ||
| M-RewardBench Reward models choose preferred responses across chat, safety, reasoning and translation tasks in 23 languages. Accuracy measures preference selection. | Cohere Labs Community | Overall chat quality | University | Not in cards | GPT-4 Turbo 83.5% | Oct 2024 | Oct 2024 | ||
| MatCha Models answer multiple-choice questions about material structure, shape and properties using images from materials characterization instruments. | Chinese University of Hong Kong, Shenzhen, Northeastern University and Zhejiang University | Lab and data work | Expert | Not in cards | | Sep 2025 | Apr 2025 | ||
| MCPTox Agents use tools whose descriptions contain untrusted instructions. The benchmark measures whether agents follow those instructions across 1,348 cases from 45 servers. | MCPTox authors | Jailbreaks and prompt injection | University | Not in cards | | Aug 2025 | Apr 2025 | ||
| MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. | Shanghai AI Laboratory and collaborators | Medicine | Expert | Not in cards | Claude Opus 4.7 69.2 | Jun 2026 | May 2026 | ||
| MedBench 4.0 (OpenCompass) Replaced in Jun 2026 | Not in cards | Nov 2025 | |||||||
| MedBench 3.0 (OpenCompass) Replaced in Nov 2025 | Not in cards | Mar 2025 | |||||||
| MedBench 2.0 (OpenCompass) Replaced in Mar 2025 | Not in cards | Jul 2024 | |||||||
| MedBench 1.0 (OpenCompass) Replaced in Jul 2024 | Not in cards | Jan 2024 | |||||||
| MultiFinBen Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score. | The Fin AI (Qianqian Xie and coauthors) | Finance | University | Not in cards | | Jun 2025 | Apr 2025 | ||
| MultiFinGround Models identify accounting concepts and extract every matching fact from English, French, and Greek financial reports. The benchmark checks their selections against annotated disclosures. | University of Manchester and collaborators | Finance | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| MyPCBench Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks. | Carnegie Mellon University | Desktop apps | Everyday | Not in cards | Claude Opus 4.8 62% | Jun 2026 | Jul 2026 | ||
| NoLiMa-Hard Models retrieve latent associations for the ten hardest NoLiMa question-needle pairs. Results are reported separately at four context lengths. | Adobe Research and LMU Munich (Modarressi et al.) | Finding facts in long inputs | Everyday | Not in cards | o3 78.5% | May 2025 | |||
| NotSoTiny 25-12 Models reconstruct a missing Verilog module from surrounding hardware design code. A formal checker compares the result with the original module. | Barcelona Supercomputing Center | Electronics | University | Not in cards | Kimi K2.5 31.6% | Dec 2025 | Apr 2026 | ||
| NPHardEval Models solve algorithmic decision and optimization problems that vary in computational complexity. | NPHardEval team | Language reasoning | University | Not in cards | GPT-4 Turbo 37.7% | Dec 2023 | Apr 2024 | ||
| OCT-Bench Models answer 10,076 questions about retinal scans, from anatomy to treatment and prognosis. The evaluator measures answer accuracy and counts invalid responses as incorrect. | Shandong University, Joint SDU-NTU Centre for Artificial Intelligence Research, and collaborators | Medicine | Expert | Not in cards | GPT-5.4 Mini 62% | Jul 2026 | Mar 2026 | ||
| OdysseyArena AI Trading Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials. | OdysseyArena authors | Puzzles and chess | Expert | Not in cards | gpt-oss-120b 23.3% | Feb 2026 | Dec 2025 | ||
| OdysseyArena Energy Dispatch Agents manage energy dispatch under hidden rules. The score is average task success over four trials. | OdysseyArena authors | Puzzles and chess | Expert | Not in cards | GPT-5 23.3% | Feb 2026 | Dec 2025 | ||
| OdysseyArena Repo System Agents operate repositories with changing hidden rules. The score is average task success over four trials. | OdysseyArena authors | Puzzles and chess | Expert | Not in cards | GPT-5 62.5% | Feb 2026 | Dec 2025 | ||
| OdysseyArena Turn on Lights Agents learn hidden switches and turn on lights. The score is average task success over four trials. | OdysseyArena authors | Puzzles and chess | Expert | Not in cards | GPT-5 28.3% | Feb 2026 | Dec 2025 | ||
| Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. | Alibaba Group (Qwen, Alibaba Token Hub) + Hong Kong Polytechnic University | Video understanding | Everyday | Not in cards | | Oct 2026 | May 2026 | ||
| OmniLayout Models place components inside a board outline. The evaluator checks overlap, routing, and electrical functionality against the schematic and the reference layout. | Pennsylvania State University and collaborators | Electronics | University | Not in cards | | Jun 2026 | Apr 2026 | ||
| Open Tabletop GM Session Command Probe Models handle five game master commands and conversational turns, including loading a campaign and rolling dice. Pattern checks classify each response. | Bobby Gray | Multi-turn conversation | Everyday | Not in cards | | Apr 2026 | Apr 2026 | ||
| OSWorld-MCP Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps. | Peking University, Tongyi Lab (Alibaba) and Beijing Zhongguancun Academy | Desktop apps | Everyday | Not in cards | Kimi K2.5 56.6% | Oct 2025 | Feb 2026 | ||
| P3D-Bench Models generate parametric 3D objects from text, images, or assembly instructions. The evaluator checks whether outputs run and preserve geometry, topology, and part structure. | Yikang Yang and colleagues | Image generation and editing | University | Not in cards | GPT-6 Astra 69.4 | Jun 2026 | Sep 2026 | ||
| PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. | Amazon | Medicine | Expert | Not in cards | Claude Opus 4.8 4.25 | Jul 2026 | Jun 2026 | ||
| PhysicalRealismBench-U VLM watches 1,300 synthetic Isaac Sim videos, flags physics-law violations, names object and frames. Scored by joint realism F1. | Reka | Physical prediction | Everyday | Not in cards | GPT-5.5 60.8% | Jun 2026 | May 2026 | ||
| PlanBench-XL Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures. | University of Illinois Urbana-Champaign | Function calling | Expert | Not in cards | | Jun 2026 | May 2026 | ||
| Portal2Bench Agents solve Portal 2 chambers using screenshots, position readings, and game controls. Each chamber counts if the agent reaches the next map within $30 and 300 plans. | Kingston Yip | Video games | Everyday | Not in cards | GPT-6.2 Astra 26.0 | Nov 2026 | Nov 2026 | ||
| PowerBench Agents retrieve equipment records and telemetry to answer 300 power-system questions. Macro Joint Accuracy averages fully correct answers equally across three task families. | OpenCompass PowerBench team | General assistants | Expert | Not in cards | GPT-5.6 Sol 74.2% | Sep 2026 | Nov 2026 | ||
| Prediction Arena Agents research events and trade contracts on Kalshi and Polymarket. The leaderboard ranks accounts by cash plus the market value of open positions. | Arcada Labs and Harvard University | Real-world events | University | Not in cards | Claude Opus 4.6 $138,945 | Jan 2026 | Mar 2026 | ||
| Preference Proxy Evaluations Reward models and judges rank responses. The benchmark compares their decisions against user preferences and verifiable answers. | Arena | Overall chat quality | University | Not in cards | Claude 3.5 Sonnet 68.5% | Oct 2024 | Sep 2024 | ||
| ProgramDistill Agents inspect working web applications and restore missing features in editable copies. Replayed browser traces check whether patches recover the reference behavior. The paper evaluates partial repairs and full application reconstruction. | Microsoft Research, Microsoft AI, and KAIST | Web and app front ends | University | Not in cards | GPT-6 Astra 84.3% | Sep 2026 | Sep 2026 | ||
| QF-Bench Agents write and run code for 87 quantitative finance tasks in a network-isolated sandbox with real data. Tasks cover derivatives pricing, fixed income, credit, risk, factor research, backtesting and market microstructure. More than 3,300 assertions check numeric outputs against finance rules such as no-arbitrage bounds and put-call parity. Scored by pass@1 over three runs. | QF-Bench team (anonymous NeurIPS 2026 submission) | Finance | Expert | Not in cards | GPT-5.5 61.7% | May 2026 | Apr 2026 | ||
| Quant-Arena Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window. | Einsia AI (Navers Lab, LaPlace's Eye) | Finance | Expert | Not in cards | GPT-6 Astra 51.3 | Oct 2026 | Sep 2026 | ||
| QuantEval Models answer quantitative finance questions, solve mathematical problems, and implement trading strategies. The authors report answer accuracy and test the generated code. | Zhaolu Kang and colleagues | Finance | University | Not in cards | Claude Sonnet 4.5 86% | Jan 2026 | Sep 2025 | ||
| Read My Aura Models respond to requests for aura readings. The tasks probe invented perceptions, flattery, and whether a change in a name or image tint changes the reading. | ashley e. romano | Wellbeing and sycophancy | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. | AgiBot | Simulated robots | Everyday | Not in cards | Claude Opus 5.5 76.3% | Jan 2026 | Oct 2026 | ||
| Role-play Benchmark (MiniMax) Role-play agents are tested on situated reenactment in Chinese and English scenes. Grading targets the immersive experience rather than a verifiable answer. | MiniMax | Creative writing | Everyday | Not in cards | | Jan 2026 | Jan 2026 | ||
| RVG-Bench Models locate the video interval that answers a question about sounds and images. The evaluator measures overlap with the annotated interval. | Yue Feng and colleagues | Audio understanding and voice | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. | AItonomy Foundation and the SLE team | Research science | Research frontier | Not in cards | GPT-6.1 Sol 44.7 | Nov 2026 | Oct 2026 | ||
| SciKnowEval Models answer scientific questions and apply knowledge in biology, chemistry, materials and physics. The suite uses automatic checks and model judges for different tasks. | Kehua Feng and colleagues | Science questions | University | Not in cards | o4-mini 0.71 | Jun 2024 | May 2025 | ||
| SheetBench Agents solve finance and accounting spreadsheet tasks through a browser in a fresh Google Sheet. The score is the share of expected answer cells that match exactly. | HUD | Finance | Expert | Not in cards | GPT-6 Astra 64.7% | Oct 2026 | |||
| Shop Assistant Honesty Bench (UA/RU) Models act as shop assistants in Ukrainian or Russian. The tests check language choice and whether quoted prices and stock match the catalog. | DimaTopka | Customer service | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. | Naïve | Simulated companies | Expert | Not in cards | GPT-6.1 Sol $2,949 | Oct 2026 | Sep 2026 | ||
| SolidWorks Bench Agents model the custom parts of 73 real machines in SolidWorks and assemble them from raster drawing packages, through screenshots and the SolidWorks API. A grader checks the geometry against the reference and reads mates, motion and feature trees from the native files. | Markov | CAD models | Expert | Not in cards | Claude Fable 5.2 0.57 | Nov 2026 | Nov 2026 | ||
| SolveEdit Models edit an image to solve a visual problem. The evaluator checks whether the goal holds and whether unrelated scene details survive the edit. | Wenjie Shu and colleagues | Image generation and editing | School | Not in cards | GPT-Image-2 57% | Sep 2026 | Feb 2026 | ||
| Spatial Reasoning: Rotating ASCII Art Models rotate binary images represented as grids of text characters. The tasks vary the grid size and compare the output with the expected rotation. | Neglectful Persimmon | Spatial reasoning and counting | School | Not in cards | Claude Sonnet 5 0.80 | Oct 2026 | Oct 2026 | ||
| SpatialBench-Long (original study) Agents complete 24 tasks in long spatial transcriptomics workflows. The authors compare their answers with results from the original analyses. | LatchBio | Genomics and life sciences | Expert | Not in cards | GPT-5.5 11.1% | May 2026 | May 2026 | ||
| SPGISpeech Models transcribe English financial recordings. The references include formatting such as capitalization and punctuation. | Kensho and collaborators | Speech recognition and translation | Everyday | Not in cards | | Apr 2021 | Oct 2026 | ||
| SPOT Models inspect published manuscripts for errors that led to corrections or retractions. A judge checks whether each reported error matches the documented error and location. | Guijin Son and colleagues | Research science | Expert | Not in cards | o3 6.10% | May 2025 | Apr 2025 | ||
| SquadStack Conversational LLM Benchmark for Voice Agents Models write the next reply or tool call from the history of an Indian outbound phone call. A language model grades each response against seven criteria. | SquadStack MLOps Team | Customer service | University | Not in cards | | Nov 2026 | Sep 2026 | ||
| SuperCLUE-OnDevice Phone-based models respond to requests in Chinese. | SuperCLUE | Overall chat quality | Everyday | Not in cards | | Aug 2026 | |||
| SWE-Clar Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build. | Upwork | Building whole projects | Expert | Not in cards | GPT-6 Astra 67% | Oct 2026 | Sep 2026 | ||
| The Aggregate Forecasting Resolved Questions Models estimate probabilities for resolved Manifold markets. The author drops questions when a model correctly recalls the outcome, then scores the remaining forecasts by squared error. | Mikhail Doroshenko (The Aggregate) | Real-world events | Everyday | Not in cards | Claude Opus 5 0.27 | Sep 2026 | Jul 2026 | ||
| ToolMaze Agents complete tool-use tasks while the runtime introduces explicit or hidden tool failures. A judge checks completion across four task structures and five perturbation settings. Metrics report task success, recovery rate, and recovery cost. | Shanghai AI Laboratory, East China Normal University, Soochow University and others | Function calling | University | Not in cards | | Jun 2026 | Apr 2026 | ||
| TraceTrial-Bench Agent harnesses, not only models, are tested on reliability across seven components, with traces used to attribute each failure to its cause. | TraceLite AI | General assistants | Expert | Not in cards | Claude Opus 5 77.6% | Oct 2026 | Aug 2026 | ||
| Unsatz Agents design experiments and propose equations for an unknown simulated system. Their final program must predict held-out interventions and conditions beyond the original experiments. | University of California, Los Angeles | Research science | Expert | Not in cards | GPT-6 Astra 60% | Oct 2026 | Sep 2026 | ||
| UserBench Agents help simulated users who start with vague goals and reveal preferences bit by bit. Agents must ask clarifying questions and use tools to reach choices that fit the user. | Salesforce AI Research | General assistants | Everyday | Not in cards | Claude Sonnet 4 0.61 | Jul 2025 | Feb 2026 | ||
| VBVR-Bench Video generation models solve visual reasoning tasks by generating a video from a starting image and instruction. Rule-based scorers check the result instead of a judge model. Tasks cover abstraction, knowledge, perception, spatial reasoning and transformation, with in-domain and out-of-domain splits. | Video-Reason (Maijunxian Wang, Ruisi Wang and others) | Video world models | Everyday | Not in cards | | Feb 2026 | Jul 2025 | ||
| VBVR-Pro The current official results table evaluates the Pro release. Original VBVR results must remain separate. | Video-Reason (Maijunxian Wang, Ruisi Wang and others) | Video world models | Everyday | Not in cards | | Feb 2026 | Jul 2025 | ||
| VerifierBench Judge models decide whether proposed answers match references in math, science, and other subjects. | OpenCompass and collaborators | Other | University | Not in cards | GPT-4.1 81.1% | Aug 2025 | Apr 2025 | ||
| ViBench 2.0 Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans. | ViBench team (Replit and collaborators) | Building whole projects | Expert | Not in cards | GPT-6.1 Sol 64.7% | May 2026 | Sep 2026 | ||
| VideoScore-Bench v2 Models score 500 generated videos on visual quality, agreement with the prompt, and physical consistency. The evaluation checks their ratings against human scores. | TIGER Lab and collaborators | Video understanding | Everyday | Not in cards | Claude Sonnet 4 28.9% | Sep 2025 | Aug 2025 | ||
| VideoScore-Bench v1 Replaced in Sep 2025 | Not in cards | | Jun 2024 | May 2024 | |||||
| Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. | Zelai Xu and colleagues | Multi-agent games | Everyday | Not in cards | o3 84.9% | Jun 2025 | Apr 2025 | ||
| VoiceCodeBench Speech-to-text models transcribe 300 English workplace recordings. They are scored on whether every target entity, such as an address or email, survives exactly. | besimple.ai (board run by Vals AI) | Speech recognition and translation | Everyday | Not in cards | GPT-Live-Transcribe 67.7% | Aug 2026 | Sep 2026 | ||
| VoxMem Audio language models answer questions that need memory of earlier spoken sessions. Questions cover what was said, who said it, how it was spoken and what was audible. | University of Melbourne | Audio understanding and voice | Everyday | Not in cards | | Sep 2026 | Sep 2026 | ||
| WARC-Bench Screenshot-only agents do short web subtasks, such as date pickers and scroll containers, on real sites replayed from Web Archive snapshots. A program checks the end state. | Uniphore | Web browsing | Everyday | Not in cards | Claude Sonnet 4 64.8% | Oct 2025 | Aug 2025 | ||
| Workspace-Bench-Lite Agents complete 100 workspace tasks. The leaderboard reports the percentage of 1,856 rubric checks passed, with separate runs for each model and harness. | SJTU + ByteDance (OpenDataBox) | Documents and spreadsheets | University | Not in cards | Claude Opus 4.7 66.7% | May 2026 | Apr 2026 | ||
| WorldCoder-Bench Agents write code that builds interactive 3D worlds. The evaluator checks whether the resulting world behaves as specified. | Shuo Lu and colleagues | Image generation and editing | University | Not in cards | GPT-5.4 27.8% | Jun 2026 | Apr 2026 | ||
| Your LLM doesn't Know BOUNDARY!. Models complete work tasks at different access levels. Tests check whether they solve the part they own and report a blocker when another owner must act. | T-RexBytes | General assistants | University | Not in cards | GPT-6.1 Sol 0.91 | Oct 2026 | Sep 2026 | ||
| ZIQI-Eval Models answer multiple-choice questions about music history, performance, and theory. A second test asks them to choose the continuation of a melody. | Jiajia Li and collaborators | Exam questions | University | Not in cards | chatglm2-6b 26.1% | Feb 2024 | Sep 2023 | ||
| ABC-Bench Coding agents take a real backend repository, edit the code, set up the environment and deploy the service in a container. The 224 tasks come from 127 open-source repositories in 8 languages and 19 frameworks. A task passes only when the running service passes external end-to-end API tests. | OpenMOSS (Fudan University) | Fixing real repositories | Expert | Not in cards | Claude Sonnet 4.5 63.2% | Jan 2026 | Dec 2025 | ||
| AD-Bench Agents answer advertising analytics requests with production-platform tools. A model judge checks final answers against refreshed reference answers and measures coverage of expert tool-call trajectories. Results report success over one or three attempts. | Tencent | Tasks from real jobs | University | Not in cards | | Feb 2026 | Dec 2025 | ||
| AdaExploreBench Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice. | HKUST (KnowComp) | Puzzles and chess | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| AfroBench Models perform language understanding, reasoning, and generation tasks across 64 African languages. The full suite draws on 22 datasets. | Jessica Ojo and colleagues | Knowledge in many languages | School | Not in cards | GPT-4o 59.6 | Nov 2023 | Sep 2024 | ||
| Agent-Diff Agents use code to complete tasks across replicas of Slack, Linear, Box, and Calendar. The evaluator compares the resulting application state with the expected changes. | Hubert M. Pysklo, Artem Zhuravel, and Patrick D. Watson | Apps and MCP servers | University | Not in cards | | Feb 2026 | Dec 2025 | ||
| Agent-SafetyBench LLM agents act in 349 tool environments over 2,000 test cases covering 8 safety risk categories and 10 failure modes. A scorer model rates each run safe or unsafe. | Tsinghua University (CoAI) | Refusing harmful requests | Everyday | Not in cards | Claude 3 Opus 59.8% | Dec 2024 | Oct 2024 | ||
| AGI-Eval Community Text to Video Models generate videos from text. The board reports objective, subjective and combined scores. | AGI-Eval (Meituan) | Video generation | Everyday | Not in cards | | Feb 2025 | Apr 2026 | ||
| AI AgriBench Smallholders Models answer questions that smallholder farmers sent to advisory services. A judge scores the responses against answers from the advisors. | AI AgriBench Consortium, University of Illinois Urbana-Champaign | Tasks from real jobs | University | Not in cards | GPT-5 mini 80.9 | Nov 2025 | |||
| AI Consumer Index Models with web search handle everyday consumer requests in shopping, food, gaming and DIY. Prompt-specific rubrics with hurdle and grounding criteria grade each answer. | Mercor | General assistants | Everyday | Not in cards | GPT-5 56.1% | Dec 2025 | Nov 2025 | ||
| AIDABench Agents analyze documents and spreadsheets to answer questions, draw charts, and generate files across 603 tasks. They can execute Python in a sandbox. Separate model evaluators check answer correctness, chart correctness and readability, and spreadsheet structure and content. | SenseTime Research and collaborators | Documents and spreadsheets | Expert | Not in cards | Claude Sonnet 4.5 59.4% | Feb 2026 | Jan 2026 | ||
| AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. | ByteDance Seed and collaborators | Scientific code | Expert | Not in cards | Claude Opus 4.5 44% | Dec 2025 | Dec 2025 | ||
| ALBA Models answer 800 European Portuguese prompts across eight areas, including wordplay, grammar and sound. A language model grades the answers. | NOVA University of Lisbon | Knowledge in many languages | School | Not in cards | | Feb 2026 | Sep 2025 | ||
| Apodex Discovery Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors. | Apodex | Research science | Research frontier | Not in cards | GPT-6 Astra 2.80 | Aug 2026 | Sep 2026 | ||
| Audio Arena Speech-to-speech models hold one long spoken conversation per scenario and call tools when needed, such as booking a dental appointment or ordering groceries. Six scenarios add up to 221 spoken turns with corrections, false memory traps, prompt injection and state tracking. An LLM judge grades each turn against a golden answer. | Arcada Labs (DesignArena) | Speech recognition and translation | Everyday | Not in cards | Grok Voice 3.0 85.7 | Feb 2026 | Oct 2026 | ||
| Autoform-Hard Agents complete Lean 4 proofs on 56 formalization tasks that earlier models failed. | Vals AI | Formal proofs | Expert | Not in cards | GPT-6.2 Astra 87.5% | Nov 2026 | Nov 2026 | ||
| Balatro Bench (Michael Andrzejewski) Agents play one fixed Balatro seed with the Red Deck on White Stake. The maker ranks runs by ante reached, breaking ties by best hand. | Michael Andrzejewski | Video games | Everyday | Not in cards | GPT-6 Astra 18.0 | Aug 2026 | Nov 2026 | ||
| BHL OCR Eval Models transcribe scanned pages from historical natural history books. | finebooks | Documents, charts and text in images | School | Not in cards | | Aug 2026 | Oct 2026 | ||
| BIG-bench Abstract Narrative Understanding Models select the English proverb that best fits a short story. Three versions offer five, ten or 100 choices. | University of North Carolina at Chapel Hill | Language reasoning | Everyday | Not in cards | | Jun 2021 | Aug 2025 | ||
| BiomniBench Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer. | Phylo, Humanlaya Data Lab and xbench (HongShan) | Genomics and life sciences | Expert | Not in cards | Claude Opus 4.7 63.9 | May 2026 | Apr 2026 | ||
| BIRD-Python Models write pandas code to answer questions about tables. A language model compares the executed output with a reference result. | Zhejiang University of Technology and University of Aberdeen | Data science and SQL | University | Not in cards | | Jan 2026 | Sep 2025 | ||
| BRoverbs Models select the Brazilian proverb that fits a story or the story that illustrates a proverb. Both directions offer five choices. | University of Campinas and Maritaca AI | Reasoning in many languages | Everyday | Not in cards | Claude 3.5 Haiku 95% | Sep 2025 | Oct 2024 | ||
| CaLM Models answer causal discovery, intervention, and counterfactual reasoning tasks in English and Chinese. Evaluation compares answers with reference labels to measure accuracy and robustness across causal targets and prompting methods. | Shanghai AI Laboratory, Tongji University, Shanghai Jiao Tong University and others | Language reasoning | University | Not in cards | Claude 2 45.3% | May 2024 | Sep 2023 | ||
| Cekura Turn Bench Mean end-of-turn score across six line conditions, as published by Cekura. | Cekura | Audio understanding and voice | Everyday | Not in cards | GPT-Realtime-2.1 84.7% | Jul 2026 | Oct 2026 | ||
| CFLUE Models answer Chinese financial questions in two parts. The knowledge part has more than 38,000 multiple-choice questions from financial qualification exams, with explanations to test reasoning. The application part has more than 16,000 test items for classification, translation, relation extraction, reading comprehension and text generation. | Alibaba Cloud (Qwen DianJin) and Soochow University | Finance | University | Not in cards | | May 2024 | Nov 2023 | ||
| CFLUE Knowledge Models answer Chinese financial qualification questions. The score is answer accuracy across the knowledge test. | Alibaba Cloud (Qwen DianJin) and Soochow University | Finance | University | Not in cards | | May 2024 | Nov 2023 | ||
| ChemDIRT Models answer chemistry questions while the evaluator varies instructions and molecular representations. The authors measure accuracy and consistency across variants. | Eric Inae and colleagues | Science questions | University | Not in cards | Claude Sonnet 5 48.7% | Mar 2026 | Jul 2026 | ||
| ChinaBench Models answer politically sensitive questions about China. An independent judge labels each response compliant, refused or evasive. | Adam Holter | Deception and alignment | Everyday | Not in cards | | Feb 2026 | Jun 2026 | ||
| CIBench Agents use code interpreters to analyze data. The evaluation checks stages of the interactive analysis. | OpenCompass and collaborators | Data science and SQL | University | Not in cards | GPT-4 Turbo 75.5% | Jul 2024 | Mar 2025 | ||
| Company Agent Bench 205 questions on how well an agent understands its company, built from a month of real Slack history: who wrote an anonymized message, whom a teammate tagged, and who sent each reply. Agents search a replayed sandbox workspace using only messages from before the answer. | Concept | Simulated companies | Everyday | Not in cards | Claude Fable 5.2 82.2% | Nov 2026 | Nov 2026 | ||
| Composite-Bench Agents solve generated browser tasks in planning, rule extraction, deduplication, routing, and information gathering. Success requires matching a certified optimal solution. A separate number-partitioning task measures solution quality rather than contributing to the success rate. | Composite | Web browsing | Expert | Not in cards | Claude Fable 5 74% | Jul 2026 | Jul 2026 | ||
| CrossWordBench Models solve generated crossword grids from text or images. The tasks vary the grid size and the share of letters already filled in. | HINT-lab and collaborators | Puzzles and chess | School | Not in cards | | Mar 2025 | Mar 2025 | ||
| CSTutorBench Models tutor students who use block-based programming to control a virtual robot. A rubric grades the feedback on 17 scenarios, including debugging and answer disclosure. | University of Illinois Urbana-Champaign | Other professions | University | Not in cards | | Jul 2026 | Apr 2026 | ||
| CulturaVivaPT Models answer multiple-choice questions about Portuguese culture, including holidays, literature, food, and sports. | AMALIA project | Knowledge in many languages | Everyday | Not in cards | | Apr 2026 | Sep 2025 | ||
| DARE-Bench Agents write code to produce predictions from structured files. The benchmark checks predictions and adherence to modeling instructions. | Snowflake and collaborators | Data science and SQL | University | Not in cards | Claude 3.7 Sonnet 49.3% | Feb 2026 | Aug 2025 | ||
| DeepRSI 15 open-ended ML tasks with a 48-hour budget each. Private verifiers score gains over baselines. Models ranked by Bradley-Terry Elo. | Hillclimb | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 1,629 | Oct 2026 | Oct 2026 | ||
| DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. | Princeton + MIT + collaborators (Battleday, Whittington, Griffiths, Tenenbaum and others) | Puzzles and chess | Everyday | Not in cards | Claude Opus 5 71.4% | Aug 2026 | Jul 2026 | ||
| DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. | Stanford University and Brown University | Genomics and life sciences | Expert | Not in cards | Claude Opus 5 40.0 | Sep 2026 | Jul 2026 | ||
| EDINET-Bench Models read Japanese financial disclosures to detect fraud, forecast earnings and identify industries. | Sakana AI | Finance | Expert | Not in cards | Claude 3.5 Sonnet (October 2024) 0.73 | Jun 2025 | Sep 2025 | ||
| Ego-MC-Bench Models watch a cook follow spoken instructions and tell them when to correct a mistake. The evaluator checks intervention timing and recipe step completion. | Qualcomm AI Research, York University, and the Vector Institute | Video understanding | Everyday | Not in cards | | Jun 2026 | Feb 2026 | ||
| EvoArena Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes. | MIT | Learning from the context | Expert | Not in cards | GPT-5.5 53.3% | Jun 2026 | Apr 2026 | ||
| EvoArena Terminal-Bench-Evo Agents solve terminal tasks that change across versions. The score is the published accuracy across versioned tasks. | MIT | Learning from the context | Expert | Not in cards | GPT-5.5 65.1% | Jun 2026 | Apr 2026 | ||
| GameASG-Bench Agents build 47 browser games from written requirements. Static checks inspect each HTML file, and browser tests check controls, gameplay and state changes. | AReaL project | Building whole projects | University | Not in cards | GPT-6 Astra 55.3% | Sep 2026 | Sep 2026 | ||
| GameEngineBench Agents edit C++ files in existing Unreal Engine 5 projects. Runtime tests check the changes, and a model audits whether the behavior meets the specification. | Nitrode and collaborators | Building whole projects | University | Not in cards | Claude Fable 5 55.5% | Jul 2026 | Jun 2026 | ||
| GEBench Image generation models create interactive GUI states from prompts. A judge scores goal, logic, consistency, interface and quality across five task types. | StepFun and collaborators | General visual questions | Everyday | Not in cards | GPT-Image-1.5 63.2 | Feb 2026 | |||
| GeoNatureAgent Benchmark Agents use a geospatial API to analyze environmental indicators in Spain and Portugal. The grader checks tool use and answers against task requirements. | Gabriel Diaz-Ireland and colleagues | Function calling | University | Not in cards | Claude Sonnet 4 60.8% | Apr 2026 | Feb 2026 | ||
| GreekBarBench Models answer Greek bar exam questions in writing. A model judge grades their use of case facts, statutory articles, and legal analysis. | Odysseas S. Chlapanis and colleagues | Law | University | Not in cards | GPT-4.1 8.35 | May 2025 | Apr 2025 | ||
| GUI Knowledge Bench Models answer 3,483 questions about interface states, actions, and procedures. They inspect screenshots and task sequences from 292 applications across six platforms. | Beijing Institute for General Artificial Intelligence and university collaborators | Finding things on screen | Everyday | Not in cards | o3 78.9% | Oct 2025 | Aug 2025 | ||
| HackBench Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices. | Meta (FAIR RAM group) | Deception and alignment | Expert | Not in cards | Claude Opus 5 64.9 | Oct 2026 | Oct 2026 | ||
| HWE-bench CAE Agents run engineering simulations, extract numerical results, and report them. A verifier repeats the extraction from the solver output and checks engineering tolerances. | svd-ai-lab | CAD models | Expert | Not in cards | GPT-5.6 Sol 0.90% | Aug 2026 | Jul 2026 | ||
| HyDRA-Bench Agents interleave database queries and document retrieval to answer questions. A model judge checks the final answer. | Snowflake and collaborators | Data science and SQL | University | Not in cards | | Aug 2026 | Nov 2025 | ||
| IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. | Shanghai Jiao Tong University, Carnegie Mellon University, and collaborators | Research science | Research frontier | Not in cards | GPT-5.5 27.3% | Jul 2026 | Apr 2026 | ||
| ImageJevBench v0.3.0 Models make decisions about images and report probabilities. This version draws 300 public and 1,200 sealed items from a new pool. | Benchmark Heaven | General visual questions | Everyday | Not in cards | GPT-6 Luna 17.3 | Oct 2026 | Sep 2026 | ||
| ImageJevBench v0.1.5 Replaced in Oct 2026 | Not in cards | GPT-6 Luna 16.1 | Oct 2026 | Sep 2026 | |||||
| Ivo-micro1 ContractBench Models review contracts and return redlines across 350 attorney-written tasks in 35 scenarios and five agreement types. Tasks cover first review and replies to a counterparty. | micro1 with Ivo | Law | Expert | Not in cards | | Oct 2026 | Sep 2026 | ||
| JudgmentBench Models produce legal work for 30 tasks. Practicing lawyers grade individual answers and compare pairs of answers. | Stanford Law School, Snorkel AI, and Harvey | Law | Expert | Not in cards | Claude Opus 4.8 89% | May 2026 | Jul 2026 | ||
| K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. | Carnegie Mellon University and collaborators | Knowledge in many languages | Expert | Not in cards | GPT-5.5 45.7% | Jun 2026 | Apr 2026 | ||
| Keep the number you wrote Models convert outdoor preferences into forecast rules. Two prompts test whether they preserve the stated numbers, units, and direction of each comparison. | Vinicius Pereira | Following precise instructions | School | Not in cards | Claude Haiku 4.5 1.00 | Oct 2026 | Sep 2026 | ||
| Kinayat Models interpret Egyptian Arabic idioms, select appropriate contexts and identify connotations. The evaluation includes separate generation and multiple-choice tasks. | Carnegie Mellon University and Mohamed bin Zayed University of Artificial Intelligence | Reasoning in many languages | Everyday | Not in cards | Claude 3.5 Sonnet 95.3% | Oct 2025 | Sep 2025 | ||
| LegalBench.PT Models answer questions about Portuguese law derived from university exams. The benchmark includes case analysis, multiple selection, and true-or-false questions. | Beatriz Canaverde and colleagues | Law | University | Not in cards | GPT-4o 85.4% | Dec 2024 | Mar 2025 | ||
| LexGenius Models answer Chinese legal questions drawn from cases and exams. The authors compare results across areas including legal reasoning, ethics, and judicial practice. | Wenjin Liu and colleagues | Law | University | Not in cards | | Nov 2025 | Apr 2025 | ||
| LexPilot Models interview a user about a Chinese civil case and predict the ruling on each claim. Paired consultations reveal the effect of missing facts. | Harbin Institute of Technology and Tsinghua University | Law | Expert | Not in cards | Claude Opus 5 72.1% | Oct 2026 | Sep 2026 | ||
| LibraryDesignBench An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity. | SprocketLab (University of Wisconsin-Madison) with Snorkel AI | Building whole projects | Expert | Not in cards | Claude Opus 5.5 48.9 | Sep 2026 | Sep 2026 | ||
| LingOly Models infer linguistic rules from examples in unfamiliar languages. They use those rules to solve linguistics olympiad problems. | Andrew M. Bean and colleagues | Reasoning in many languages | Competition | Not in cards | Claude 3 Opus 46.3% | Jun 2024 | May 2024 | ||
| LiveMathBench Models solve recent competition math problems, sampled many times. The G-Pass@k metric rewards both peak accuracy and stable answers across samples. | OpenCompass (Shanghai AI Laboratory) | Competition problems with answers | Competition | Not in cards | o3-mini 76.8% | Dec 2024 | Jan 2025 | ||
| LiveResearchBench Deep research agents write cited reports for user-centric questions that need current information from many live web sources. Rubric-based judges score the reports. | Salesforce AI Research | Deep research | Expert | Not in cards | GPT-5 73.7 | Oct 2025 | Aug 2025 | ||
| LLM SoccerArena World Cup 2026 Models forecast World Cup match scores and outcome probabilities before kickoff. The organizers compare forecasts with match results under different prompts and web access rules. | LMU Munich, University of Cologne, and Paderborn University | Real-world events | Everyday | Not in cards | GPT-5.5 168.0 | Jun 2026 | Jun 2026 | ||
| MarineEval Models answer 2,000 questions about marine images and documents. The authors combine answer checks with model judgments. They report total accuracy and mean accuracy across seven tasks. | Hong Kong University of Science and Technology | General visual questions | University | Not in cards | Grok 2 Vision (1212) 50.4% | Nov 2025 | Apr 2025 | ||
| MasakhaPOS Models assign parts of speech to words in African languages. The authors compare tagging models across languages. | Cheikh M. Bamba Dione and colleagues | Reading in many languages | School | Not in cards | GPT-4o 62.8% | May 2023 | Aug 2024 | ||
| MastermindEval Models infer a hidden symbol sequence from previous guesses and feedback in Mastermind. The evaluation varies sequence length and the similarity of answer choices. | Jonas Golde and colleagues | Puzzles and chess | School | Not in cards | | Mar 2025 | Jan 2025 | ||
| MedAgentBench Agents use FHIR APIs in a virtual electronic health record to do 300 patient tasks in 10 categories written by physicians. | Stanford University | Medicine | Expert | Not in cards | Claude 3.5 Sonnet (October 2024) 69.7% | Jan 2025 | Feb 2025 | ||
| Medical Bench Agents read longitudinal dermatology charts and write clinical assessments. Expert-authored rubrics grade their findings and management decisions. | Turing | Medicine | Expert | Not in cards | Grok 4.6 88% | Sep 2026 | Aug 2026 | ||
| MentalBench-100k Models answer counseling questions. Human reviewers grade responses to 1,000 of the 10,000 prompts on seven attributes, each from one to five. | Abeer Badawi and colleagues | Medicine | Expert | Not in cards | GPT-4o 5 | Oct 2025 | Apr 2025 | ||
| MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. | Alibaba | Simulated companies | Expert | Not in cards | | Jul 2026 | Jul 2026 | ||
| MISHAP-Bench Models answer open-ended questions about audio clips. A model judge uses human annotations to identify claims the recording does not support. | Zhi Wen Soi and colleagues | Audio understanding and voice | Everyday | Not in cards | Kimi-Audio-7B-Instruct 32.8% | Sep 2026 | Aug 2026 | ||
| MJ-Bench Models compare image pairs for text alignment, safety, visual quality, and bias. Scores measure preference accuracy, with separate dispersion and equality measures for bias. | MJ-Bench team at UNC-Chapel Hill, University of Chicago, Stanford University, and collaborators | Image generation and editing | Everyday | Not in cards | InternVL-Chat-V1-5 44.5% | Jul 2024 | Apr 2024 | ||
| MLGym-Bench Agents work on 13 open-ended AI research tasks in vision, language, reinforcement learning and game theory. Scores compare the agent's result with baselines on each task's metric. | Meta | Machine learning engineering | Expert | Not in cards | | Feb 2025 | Jun 2025 | ||
| MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. | University of Illinois Urbana-Champaign (ASTRAL Group), University of Washington and UC San Diego | Deception and alignment | Expert | Not in cards | gpt-oss-120b 91.9% | Mar 2026 | Dec 2025 | ||
| MusicTheoryBench Models answer college music theory questions about musical knowledge and reasoning. Some questions include a score written in ABC notation. | Multimodal Art Projection and the ChatMusician team | Exam questions | University | Not in cards | Claude Sonnet 4.6 71.1% | Dec 2023 | Mar 2026 | ||
| NewsBench (Forum AI) Models answer questions about current events. Judges calibrated against expert ratings score factual accuracy, neutrality, and source quality. | Forum AI | Long-form factuality | Everyday | Not in cards | GPT-5.6 Terra 88.3% | May 2026 | Aug 2026 | ||
| Nuclear Decision-Making Benchmark Models answer policy questions framed from different countries’ perspectives. The authors compare response distributions and agreement across repeated runs. | Scale AI and policy researchers | Bias, fairness and politics | Expert | Not in cards | | Aug 2026 | Feb 2026 | ||
| OAB-Bench Models write answers for the second phase of the Brazilian bar examination. The port averages model judges’ scores against grading guidelines. | Maritaca AI | Law | Expert | Not in cards | | Apr 2025 | Mar 2026 | ||
| OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. | Shanghai Jiao Tong University, Pennsylvania State University, and collaborators | CAD models | University | Not in cards | | Aug 2026 | Jun 2026 | ||
| OmniGUI Models use screenshots, sound, and video to predict the next action in 29 phone apps. The grader compares each action with a human demonstration. | XPeng Motors | Phones | Everyday | Not in cards | | Mar 2026 | Feb 2026 | ||
| OmniMech Models write CAD programs to reconstruct parts from dimensioned drawings. The evaluator compares the reconstructed geometry with reference parts and measures relative volume error. | Pennsylvania State University and collaborators | CAD models | University | Not in cards | GPT-5.5 15.9% | Aug 2026 | Jun 2026 | ||
| OmniSch Models convert circuit schematics into graphs of components, pins, and connections. A second task lets models search schematic images for individual features. | Pennsylvania State University and collaborators | Electronics | University | Not in cards | | Mar 2026 | Feb 2026 | ||
| OvertonBench Models answer questions about contested issues. Participants rate how well each answer represents their viewpoint. The score measures the fraction of opinion groups represented. | Elinor Poole-Dayan and colleagues | Bias, fairness and politics | Everyday | Not in cards | | Sep 2025 | May 2025 | ||
| PaintBench Image editing models do precise single-answer edits in 20 operations such as geometric transforms, color changes and symbolic edits. Procedural generation gives deterministic checks. | VISIONx, New York University | General visual questions | Everyday | Not in cards | | May 2026 | Jun 2026 | ||
| Panda-CVL Models accept correct image-grounded responses or locate and replace their first erroneous text span. The test expands 652 conversations into 2,126 instances. Its overall F1 combines correct acceptance with exact correction of faulty responses. | StepFun and Xiamen University | General visual questions | University | Not in cards | GPT-5.5 17.1% | Sep 2026 | Sep 2026 | ||
| PaperScope Agents search multiple scientific papers to compare figures, summarize results, and solve problems. Each question can require evidence from several documents. | Lei Xiong and colleagues | Deep research | Expert | Not in cards | Grok 4 41.0 | Jan 2026 | Nov 2025 | ||
| PCBBench (pcbbench.org) Agents solve electronics problems and produce KiCad files. The public pilot checks artifacts across ten suites, including schematic design, board layout, and debugging. | ProtoFlow | Electronics | Expert | Not in cards | Claude Opus 5.5 89.6% | Oct 2026 | Oct 2026 | ||
| PE Civil Bench Models solve 110 multiple-choice and 40 numerical civil engineering exam problems. A model compares each answer with the reference solution. | University of Houston | Exam questions | University | Not in cards | GPT-5.5 87.3% | Mar 2026 | Apr 2026 | ||
| Periapical Lesions Bench Agents locate and grade lesions around tooth roots in panoramic dental X-rays. A perfect report matches every lesion and grade without adding extra boxes. | Dylan Tirandaz | Medicine | Expert | Not in cards | | Sep 2026 | Nov 2026 | ||
| PReMind-Bench Models inspect a scene and decide whether to retrieve a personal memory before answering. Judges check whether the answer addresses the hazard and uses the right person’s information. | Seoul National University | Wellbeing and sycophancy | Everyday | Not in cards | Claude Opus 5.5 87.3% | Nov 2026 | Sep 2026 | ||
| PRISM-Bench (image generation) Models generate images from English or Chinese prompts. Vision-language judges score visual appeal and prompt alignment across seven tracks, including composition, lettering, and style. | The Chinese University of Hong Kong, HKU, Beihang University, Alibaba, and SenseTime | Image generation and editing | Everyday | Not in cards | GPT-Image-1 86.3 | Sep 2025 | Sep 2025 | ||
| Prompt-to-Bench HO-16 Models design 16 additional laboratory fixtures in OpenSCAD. The grader checks each rendered part against the dimensions and features in its prompt. | S. S. Cocioba, Binomica Labs | CAD models | University | Not in cards | Claude Opus 5.5 16.0 | Oct 2026 | Sep 2026 | ||
| pyATS-Bench Models generate pyATS scripts for wireless network test plans covering access point registration, WLAN status, and client association. | Nikhil Shenoy | Function-level problems | University | Not in cards | GPT-6 Astra 1.00 | Nov 2026 | Sep 2026 | ||
| R-ConstraintBench Models produce feasible operational plans as constraints accumulate. The score gives more weight to harder levels. | Labelbox | Running a business | Expert | Not in cards | GPT-5 79.2% | Aug 2025 | Aug 2025 | ||
| Reasoning-Bias-Reporting-Quality Models respond to everyday requests that contain flawed reasoning. A judge scores whether the reply challenges the assumption without an explicit request for critique. | Gayathri Arvind | Wellbeing and sycophancy | Everyday | Not in cards | Claude Opus 5.5 6.94 | Nov 2026 | Sep 2026 | ||
| Robots.txt Benchmark Models interpret robots.txt rules from synthetic and real websites. They also write rules from policies, which the grader checks against URL probes. | Old Jard | Function-level problems | University | Not in cards | Claude Sonnet 5.5 1.00 | Oct 2026 | Sep 2026 | ||
| RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. | MAC-AutoML (Xiamen University) | Spatial reasoning and counting | Everyday | Not in cards | GPT-5.6 Sol 17% | Oct 2026 | Sep 2026 | ||
| RSI Bench Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks. | Scale AI | AI research tasks | Research frontier | Not in cards | GPT-6 Astra 38.9 | Aug 2026 | Sep 2026 | ||
| RTL-Repo Models predict the next line of Verilog using code from the same project. The authors measure exact matches and edit similarity. | Ahmed Allam and Mohamed Shalan | Electronics | University | Not in cards | | Mar 2024 | Sep 2025 | ||
| SAS-Bench Models grade short student answers from subject exams. The benchmark compares their scores and error classifications with expert annotations. | Peichao Lai and colleagues | Other professions | University | Not in cards | | May 2025 | Apr 2025 | ||
| SEAL Adversarial Robustness Models receive 1,000 single-turn adversarial prompts written by 10 full-time red teamers across nine harm categories. Safety specialists grade responses, and models are ranked by the number of high-harm violations. | Scale AI | Jailbreaks and prompt injection | Expert | Not in cards | | Jul 2024 | |||
| Speko Speech-to-Speech Benchmark (20-turn concierge call) Voice agents try to book a flight, handle a failed booking, and remember a changed date. Eight automated checks score their actions and replies. | Speko | Customer service | Everyday | Not in cards | gpt-realtime 1.00 | Jul 2026 | Jul 2026 | ||
| Speko Speech-to-Speech Benchmark (10-turn call) Replaced in Jul 2026 | Not in cards | | Jul 2026 | Jul 2026 | |||||
| SplatBench A model rebuilds an object in 3D from eight photos by placing up to 300 Gaussians itself through tool calls, with no optimizer. Half of the 30 tasks start from an empty scene, and half ask it to reassemble a scrambled fit. SplatScore compares renders from eight angles the model never sees, with an empty scene at 0 and gradient descent at 100. | Perpetual Research | Spatial reasoning and counting | University | Not in cards | GPT-6 Astra 49.6 | Oct 2026 | Oct 2026 | ||
| STAGE Cross-Scene Narrative Evolution Reasoning Models answer questions that require evidence from different scenes in a movie. A judge compares the answer with a reference. | Southeast University, Beijing Zhongguancun Academy, Nanjing Normal University and ZhuiWen Technology | Reasoning over long documents | School | Not in cards | Claude Sonnet 4.6 65.1% | May 2026 | Apr 2026 | ||
| SuperCLUE-Code3 Models write executable Python functions from Chinese specifications. Unit tests check functionality, and the leaderboard reports results by difficulty. | SuperCLUE team | Function-level problems | School | Not in cards | GPT-4 Turbo (Jan 2024) 68% | Feb 2024 | Jan 2024 | ||
| SuperCLUE-DeepSearch Agents search for information to answer Chinese research questions. | SuperCLUE | Questions that need search | University | Not in cards | ChatGPT agent 74.3 | Nov 2025 | |||
| SuperCLUE-Faith Models produce answers that remain faithful to a supplied source. | SuperCLUE | Faithful to sources | School | Not in cards | GPT-5.2 93.2 | Dec 2025 | |||
| SuperCLUE-Mkt Models complete tasks in advertising and marketing. | SuperCLUE | Running a business | University | Not in cards | | Dec 2025 | |||
| SuperCLUE-OpenClaw Models complete coding, memory, and writing tasks in the OpenClaw environment. | SuperCLUE | General assistants | University | Not in cards | Claude Opus 4.6 92.3 | Mar 2026 | |||
| SuperCLUE-SWE with mini-swe-agent Models solve Chinese software engineering tasks with the mini-swe-agent harness. | SuperCLUE | Fixing real repositories | Expert | Not in cards | | Nov 2026 | |||
| SWE-Marathon-Ext Agents build 12 software products from specifications. Automated tests and browser checks grade the applications. | Mercor | Building whole projects | Expert | Not in cards | GPT-5.6 Sol 74% | Aug 2026 | Jul 2026 | ||
| Synthetic Sheet Music Reasoning Benchmark Models answer questions about rhythm, chords, intervals, and scales. Each test question appears in text notation and as an image. | Zhilin Wang, Zhe Yang, and collaborators | Documents, charts and text in images | School | Not in cards | Claude Opus 4.6 90.5% | Sep 2025 | Mar 2026 | ||
| Table Judge Benchmark Vision language models judge whether rendered table images match candidate HTML. Controlled corruptions test false positives, true positives and injected-error detection. | Reducto | Documents, charts and text in images | Expert | Not in cards | Claude Opus 4.8 94.9% | Nov 2025 | Jul 2026 | ||
| TEM-VQA Models give short written answers about structures, image features and experimental design in transmission electron micrographs. | National Cheng Kung University | Lab and data work | Expert | Not in cards | GPT-4o 40.4 | Jun 2026 | Sep 2025 | ||
| Terminal-Bench Math Modeling Agents use a terminal to solve 18 mathematical modeling problems and submit numerical answers. Feasibility checks and numerical scoring compare answers with reference solutions. | TB-MathModel authors | Scientific code | Competition | Not in cards | GPT-5.5 64.3 | Nov 2026 | Sep 2026 | ||
| TerminalWorld Agents complete terminal workflows reconstructed from public recordings in isolated environments. Automated tests check the resulting files and system state. A manually reviewed 200-task subset supports the published model comparisons. | EuniAI and the TerminalWorld authors | Terminal tasks | University | Not in cards | Claude Opus 4.7 62.5% | May 2026 | Apr 2026 | ||
| TimeScope Models answer questions about short clips inserted into videos lasting up to eight hours. Accuracy measures retrieval, synthesis and temporal perception. | Stanford University and Hugging Face | Long videos | Everyday | Not in cards | | Jul 2025 | Jun 2025 | ||
| VAREX-Bench Models extract structured fields from 1,777 government forms using a supplied JSON schema. Normalized exact match checks field values, while string similarity gives partial credit. Array entries use order-independent matching. | IBM Research | Documents, charts and text in images | Everyday | Not in cards | | Mar 2026 | Dec 2025 | ||
| VDR-Bench Models answer 2,000 visual questions that require finding entities in images and researching related facts. A Qwen3-VL judge checks final answers against references. Entity recall separately measures whether searches retrieve the relevant entities. | CUHK MMLab, Xiaohongshu, and university collaborators | Deep research | University | Not in cards | Claude 3.7 Sonnet 27.2% | Feb 2026 | Oct 2025 | ||
| VL-RewardBench Vision-language reward models pick the better of two responses on general queries, hallucination detection and reasoning tasks. | HKU and collaborators | General visual questions | Expert | Not in cards | | Nov 2024 | Sep 2024 | ||
| VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. | PKU-Alignment | Simulated robots | Everyday | Not in cards | | Sep 2025 | May 2026 | ||
| VoxelCodeBench Models write code to build and modify voxel scenes. The benchmark checks the constructed shapes and evaluates their appearance. | Yan Zheng and colleagues | Image generation and editing | University | Not in cards | GPT-5 87.9% | Apr 2026 | Sep 2025 | ||
| WebsiteBench Agents explore reference websites and rebuild their appearance, behavior, and persistent state. Hidden browser tests require functional correctness and screenshot similarity, then report the fraction passed. | TIGER-AI-Lab | Web and app front ends | Expert | Not in cards | Claude Opus 5.5 26.2% | Sep 2026 | |||
| xbench-Recruitment Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5. | xbench (HongShan) | Running a business | Expert | Not in cards | o3 78.5 | Jun 2025 | May 2025 | ||
| A^3-Bench Models solve 2,198 science reasoning problems annotated with the anchors and attractors (prior knowledge) they need. An AAUI metric measures how much of that knowledge the model activates. | Jian Zhang, Yu He, Jun Liu et al. | Science questions | University | Not in cards | Grok 4 Fast 65.1% | Jan 2026 | Dec 2025 | ||
| AARRI-Bench Agents complete research intern tasks that test scientific judgment, professional conduct, and attention to detail. The Harbor collection contains 82 tasks. | AARR-Bench team | AI research tasks | Expert | Not in cards | Claude Opus 4.7 68.3% | Apr 2026 | Apr 2026 | ||
| ABACUS Models add, remove, or change a specified number of objects in an image. The evaluator checks the count and preservation of the surrounding scene. | Aymen Lassoued and colleagues | Image generation and editing | School | Not in cards | | Oct 2026 | |||
| Acme Agent Evals Agents use GitHub tools to answer questions about issues, pull requests, and code in a simulated software project. | Arize AI | Apps and MCP servers | University | Not in cards | Claude Sonnet 4.6 85.1% | May 2026 | Apr 2026 | ||
| AD4Edu-Bench Models write audio descriptions for university lectures. A rubric checks accessibility rules, and the authors compare the rankings with blind and low-vision listeners' preferences. | University of Zurich and collaborators | Video captioning | University | Not in cards | | Oct 2026 | Oct 2025 | ||
| ADeptS-Bench Desktop Agents choose safe interface actions on the Desktop test set. The published safety score combines task success and attack resistance. | Chen et al. | Finding things on screen | Expert | Not in cards | | Aug 2026 | Apr 2026 | ||
| ADeptS-Bench Mobile Agents choose safe interface actions on the Mobile test set. The published safety score combines task success and attack resistance. | Chen et al. | Finding things on screen | Expert | Not in cards | | Aug 2026 | Apr 2026 | ||
| Agentic Credit Analysis Models extract and standardize income statements and balance sheets. Ramp compares their figures with credit analysts’ figures using a five percent tolerance. | Ramp Labs | Finance | Expert | Not in cards | Claude Sonnet 5.5 80.2% | Oct 2026 | Sep 2026 | ||
| AI AgriBench Post-Cutoff Models answer 146 questions from newer agricultural extension documents. Judges score their answers for accuracy, relevance, completeness and conciseness. | AI AgriBench Consortium, University of Illinois Urbana-Champaign | Tasks from real jobs | University | Not in cards | GPT-5 98.1 | Jan 2026 | Nov 2025 | ||
| ANAH Models detect unsupported claims in answers to English and Chinese questions. The benchmark provides annotations for individual claims. | OpenCompass and collaborators | Hallucination rates | University | Not in cards | | May 2024 | Sep 2023 | ||
| ARE-bench Agents continue an existing research project from a real user request. A judge checks the deliverable against the request and earlier workspace requirements. | Shanghai Artificial Intelligence Laboratory and collaborators | AI research tasks | Expert | Not in cards | GPT-6.1 Sol 66.2% | Nov 2026 | Sep 2026 | ||
| Arena Search Factuality Arena combines user preferences with independent checks of verifiable claims in model responses. | Arena | Long-form factuality | Everyday | Not in cards | Grok 4.20 1,218 | Jul 2026 | Jun 2026 | ||
| AscendGYM Agents optimize existing kernels for Ascend processors using tensor shapes from real workloads. The benchmark measures speedup and token consumption. | AscendGYM team | GPU kernels | Expert | Not in cards | Claude Opus 5 74.0 | Nov 2026 | Aug 2026 | ||
| AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. | Microsoft Research | Simulated robots | Expert | Not in cards | Claude Opus 4.5 76.5% | Mar 2026 | Jan 2026 | ||
| AstroVisBench Models write code to process astronomical observations and plot the results. The benchmark compares computed values with reference outputs and uses a model to grade plots. | University of Texas at Austin and collaborators | Lab and data work | University | Not in cards | Claude Opus 4 49.8% | May 2025 | May 2025 | ||
| Baba Is AI Models see a puzzle and write a plan to reach the winning object. Some puzzles require them to push words together to change the rules. | MIT and University of Virginia | Puzzles and chess | Everyday | Not in cards | | Jul 2024 | Mar 2025 | ||
| BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. | OpenAI and Ligo Biosciences | Genomics and life sciences | Expert | Not in cards | GPT-6 Astra 66.7% | Nov 2026 | Sep 2026 | ||
| BrowserART Browser agents receive 100 harmful requests on synthetic and real websites. The test measures how often the agent attempts the harmful behavior that its backbone model would refuse in chat. | Scale AI and collaborators | Refusing harmful requests | Expert | Not in cards | Claude 3 Opus 11% | Oct 2024 | Sep 2024 | ||
| CascadeBench Agents predict the cascade of changes that one action sets off in an enterprise system, where business rules differ between deployments. The environments are synthetic and follow the World of Workflows method. Tests include shifts in the system's rules to check whether a model reads the active configuration or relies on what it learned in training. | ServiceNow | Interactive world models | Expert | Not in cards | GPT-5 41.8% | May 2026 | Apr 2026 | ||
| CASTLE Models read 250 hand-written compilable C programs and say whether each has a vulnerability and which CWE. Static analyzers run on the same set. | CASTLE authors (Richard et al.) | Finding and exploiting real bugs | Expert | Not in cards | o3-mini 977.0 | Mar 2025 | Jan 2025 | ||
| Cekura Agent Performance Scenario reliability across three repeated calls (Pass³). | Cekura | Audio understanding and voice | Everyday | Not in cards | Grok Voice 3.0 81.7% | Jul 2026 | Oct 2026 | ||
| Cekura Real Human Calls Real caller reliability across three repeated calls (Pass³). | Cekura | Audio understanding and voice | Everyday | Not in cards | Grok Voice 3.0 42.9% | Jul 2026 | Oct 2026 | ||
| ChemInsight Models solve related chemistry problems in bundles. The authors score how many parts a model can solve and how far its reasoning remains correct. | Changyu Hu and colleagues | Science questions | University | Not in cards | | Nov 2026 | Dec 2025 | ||
| CLadder Models answer yes or no to questions about associations, interventions, and counterfactuals. A causal inference engine supplies the reference answers. | CausalNLP and collaborators | Language reasoning | University | Not in cards | GPT-4 Turbo 70.4% | May 2023 | Nov 2023 | ||
| Closed-Loop Evaluation of LLM Agents for Embedded Software Development Agents write firmware from engineering briefs and test it on simulated equipment. Hidden checks score device behavior across five control tasks. | Jorge García-Carrasco and colleagues | Electronics | University | Not in cards | GPT-5.4 100% | Mar 2026 | |||
| CocoaBench Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output. | UC San Diego | Web browsing | Expert | Not in cards | GPT-5.4 45.1% | Apr 2026 | Mar 2026 | ||
| CodeCap-Bench Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail. | Kling AI (Kuaishou) + Nanjing University | Video captioning | Everyday | Not in cards | GPT-6 Astra 73.2 | Oct 2026 | Sep 2026 | ||
| ContextASR-Bench Models transcribe speech with optional domain labels and named entities. The benchmark measures transcription errors and recognition of named entities. | He Wang and colleagues | Speech recognition and translation | Everyday | Not in cards | Kimi-Audio-7B-Instruct 3.10% | Jul 2025 | Apr 2025 | ||
| CrochetBench Models identify stitches, select instructions that match a photo, write crochet patterns, and translate patterns into programs for a crochet simulator. | University of Notre Dame | General visual questions | University | Not in cards | Claude Sonnet 4 60.9 | Nov 2025 | Sep 2025 | ||
| CRUST-bench Agents translate 100 C projects into safe Rust. Tests check whether each translation matches the supplied Rust interface and preserves the original behavior. | Anirudh Khatry and colleagues | Fixing real repositories | Expert | Not in cards | o3 48% | Apr 2025 | May 2025 | ||
| CUSP Models forecast scientific progress across 4,760 events in many fields, using only knowledge from before a set cutoff date. Tasks ask whether an advance is feasible, how it would work, how to design a solution and when it will happen. The benchmark also compares events before and after each model's training cutoff. | University of Oxford and coauthors | Real-world events | Expert | Not in cards | GPT-5.6 Sol 60.4% | May 2026 | Jul 2026 | ||
| DailyReport Search agents answer practical daily information requests. Evaluators assess the resulting reports across dimensions of user satisfaction. | AGI-Eval (Meituan) | Deep research | Everyday | Not in cards | GPT-5.4 2.89 | Jun 2026 | Mar 2026 | ||
| DPAI Arena Spring Benchmark Coding agents implement changes in Java Spring repositories. DPAI Arena scores each run and publishes its duration. | JetBrains and Toloka | Fixing real repositories | Expert | Not in cards | Claude Opus 4.7 64.9 | Oct 2025 | Apr 2026 | ||
| DriveBench Vision-language models answer 20,498 questions about 19,200 driving frames, covering perception, prediction, planning and behavior. Inputs come in 17 settings: clean images, corrupted images and text only. The test checks whether answers rest on the image or on general knowledge. | UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU | Embodied reasoning | Everyday | Not in cards | Qwen2-VL-72B-Instruct 48.0 | Jan 2025 | Aug 2024 | ||
| EgoBlind Models answer questions from blind users using only video recorded before each question. GPT-4o mini checks each answer against human references. | National University of Singapore and collaborators | Video understanding | Everyday | Not in cards | GPT-4o 59.3% | Mar 2025 | Sep 2025 | ||
| ELBench Models respond to teaching and safety scenarios alongside academic questions. The authors compare the same models across four educational assessment areas. | Yilin Jiang and colleagues | Other professions | University | Not in cards | | Jun 2026 | May 2026 | ||
| EmailBench Agents handle email and productivity requests in a synthetic inbox. The evaluator combines executable checks and model-graded rubrics across 206 scenarios. | Microsoft | Documents and spreadsheets | University | Not in cards | Claude Sonnet 4.5 33.5% | Sep 2026 | Apr 2026 | ||
| Embodied Over-Refusal Benchmark Models decide whether to perform household tasks. The score balances compliance on safe requests with refusal on hazardous ones. | sky | Refusing harmful requests | Everyday | Not in cards | GPT-5.6 Sol 1.00 | Nov 2026 | Oct 2026 | ||
| EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. | Hongcheng Gao and colleagues | Simulated robots | University | Not in cards | Claude Opus 5.5 41.4% | Oct 2026 | Sep 2026 | ||
| EngiWorld Agents work through the full industrial engineering design loop on 1,301 expert-curated tasks that need reasoning over geometric and physical constraints across software. | Zhiman AI | CAD models | Expert | Not in cards | Claude Opus 5 44.7% | Sep 2026 | Sep 2026 | ||
| EnvSimBench Language models act as simulators of tool environments for agent training. Given a trajectory of agent actions, the model predicts the next environment state and feedback. Labels come from real executors, so the score checks whether simulated states match what the real environment would return. | EnvSimBench authors | Apps and MCP servers | University | Not in cards | | May 2026 | Mar 2026 | ||
| EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. | Carnegie Mellon University + Stanford + collaborators | Research science | Research frontier | Not in cards | Claude Fable 5.1 29.8 | Sep 2026 | Sep 2026 | ||
| Evaluating AI Safety in Teen Conversations Chatbots respond to simulated teenagers across ten replies. Judges measure the share of conversations that fail a critical safety check. | Vals AI | Wellbeing and sycophancy | University | Not in cards | | Sep 2026 | Aug 2026 | ||
| EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. | OpenAI and Paradigm | Finding and exploiting real bugs | Expert | Not in cards | GPT-5.3 Codex 51% | Feb 2026 | Feb 2026 | ||
| ExpertCrowd-SWE Agents resolve 100 software issues across 87 repositories in 11 languages. Each engineer-written rubric combines behavioral tests with a model judge and awards partial credit. The score averages five valid runs per task and then averages tasks. | Toptal ExpertCrowd | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 67.3% | Oct 2026 | Sep 2026 | ||
| ExpertEval Models answer cases in medicine, finance and law. Practicing professionals write the cases and supply weighted grading rubrics. | UniPat AI | Other professions | Expert | Not in cards | Claude Opus 4.6 66.9% | May 2026 | Mar 2026 | ||
| ExtractBench Pro Systems extract every schema value from long-tail synthetic documents with charts, drawn relations, house rules and jargon. Scored by value F1 and by grounded F1, which also needs the source location. | LlamaIndex | Documents, charts and text in images | Expert | Not in cards | GPT-6 Astra 58% | Oct 2026 | Oct 2026 | ||
| Far-Field ASR Leaderboard Models transcribe speech under simulated room acoustics. The board compares word error rates. | Treble Technologies and Hugging Face | Speech recognition and translation | Everyday | Not in cards | | Jun 2026 | Oct 2026 | ||
| FinDeepIndicator Agents construct financial indicators by specifying formulas, retrieving market data, and calculating results. The evaluation checks performance at each step. | Chaoqun Yang and colleagues | Finance | University | Not in cards | | Aug 2026 | Apr 2026 | ||
| FinEval Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings. | Shanghai University of Finance and Economics (AIFLM Lab) | Finance | University | Not in cards | | Aug 2023 | Sep 2024 | ||
| FinMaster Models complete financial workflows using simulated company records. The benchmark includes single calculations and tasks that require several financial metrics. | Junzhe Jiang and colleagues | Finance | University | Not in cards | o3-mini 73% | May 2025 | Apr 2025 | ||
| FinToolBench Agents answer 295 finance questions by selecting and executing tools from a library of 760 financial interfaces. The evaluation separates tool execution, answer correctness, and mismatches in timeliness, intent, and regulatory domain. Answer checks combine model judging with exact matching. | Jiaxuan Lu, Kong Wang, and collaborators | Function calling | Expert | Not in cards | | Mar 2026 | Mar 2026 | ||
| Frontier Agentic ML & DS Agents solve machine learning and data science tasks in a terminal. Verifiers check their deliverables and predictive quality. | Turing | Machine learning engineering | Expert | Not in cards | Claude Opus 5 64.6% | Aug 2026 | Jul 2026 | ||
| FrontierCAD Assemblies Agents place supplied CAD parts into 16 assemblies. The score is the published mean assembly geometry score. | Normal Research | CAD models | Expert | Not in cards | GPT-6 Astra 0.71 | Oct 2026 | Sep 2026 | ||
| FrontierCAD Parts Agents build 39 CAD parts from engineering drawings. The score is the published mean geometry score. | Normal Research | CAD models | Expert | Not in cards | GPT-6 Astra 0.63 | Oct 2026 | Sep 2026 | ||
| FrontierMemory A model reads 750 past agent trajectories and writes memory for a task agent. The score is how many new coding and knowledge-work tasks that agent then solves. | Letta | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 73.3% | Oct 2026 | |||
| FutureX Agents predict upcoming events collected daily from 195 websites in 11 domains, within a one-week window. Answers are scored once real results are crawled, with harder tiers weighted more. | ByteDance Seed with Fudan, Stanford and Princeton | Real-world events | Expert | Not in cards | Grok 4 27.9 | Aug 2025 | Jul 2025 | ||
| Gaia2-CLI Agents complete realistic tasks through command-line tools. The benchmark measures task success across search, execution, adaptability, ambiguity, and time constraints. | Meta FAIR | General assistants | Everyday | Not in cards | Claude Opus 4.6 57% | Apr 2026 | Apr 2026 | ||
| Garble-Bench (Voice Arena) Models respond to dialogue with speech recognition errors. The evaluation measures error detection, transcription fidelity, conversation recovery and task completion. | Voice Arena and collaborators | Audio understanding and voice | Everyday | Not in cards | | Sep 2026 | Sep 2026 | ||
| Groundtruth Dynamic Benchmarking (Geology Edition) Models answer questions from geological reports. A judge scores each answer from zero to ten against a rubric tied to passages in the source documents. | Eigenform | Reading comprehension | Expert | Not in cards | Kimi K3 8.81 | Jul 2026 | Aug 2026 | ||
| HarnessAudit-Bench Agents run inside execution harnesses such as Claude Code, Codex and OpenClaw on 210 tasks across eight domains, from finance to software engineering, with safety rules built in. The audit reads the full trajectory through channels the agent cannot change. It scores whether the agent stays inside its permission and information-flow policies, completes checkpoints, and keeps both under prompt injection, vague goals and tool errors. | UC Santa Barbara, with UC Berkeley, UW-Madison, Stanford and Microsoft Research | Refusing harmful requests | Expert | Not in cards | | May 2026 | Apr 2026 | ||
| HCSU-Bench Models match a Chinese calligraphy sample to one of eight styles. A second task asks models to describe the brushwork in fewer than 50 words. | Tongji University | General visual questions | University | Not in cards | GPT-5.2 34.6% | Mar 2026 | Dec 2025 | ||
| HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. | Meta, Nanyang Technological University, University of Washington, Brown University, and Northwestern University | Simulated robots | Everyday | Not in cards | | Jul 2026 | Sep 2026 | ||
| Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. | Federico Ravenda and colleagues | Medicine | Expert | Not in cards | GPT-6 Luna 83.9% | Nov 2026 | Sep 2026 | ||
| Hype Trap Models assess anonymized news about small stocks. The grader compares their predictions with stock movements over the next five trading days. | SaltySteve | Real-world events | University | Not in cards | Claude Haiku 4.5 0.01 | Oct 2026 | Sep 2026 | ||
| Invented O'Clock Models calculate dates and times in 45 scheduling problems. Paired versions remove an essential fact and require the model to answer unknown. | Maddie Brooks | Faithful to sources | School | Not in cards | | Oct 2026 | Apr 2026 | ||
| IPI Bench Agents complete workplace workflows while attackers add misleading records to an emulated company with 11 services. The benchmark pairs six workflow types with harmful injected objectives across 34 tasks. The headline here measures single-query attack success as a percentage, where lower is better. | General Analysis | Jailbreaks and prompt injection | Expert | Not in cards | Claude Opus 5.5 4.80% | Oct 2026 | Oct 2026 | ||
| IslamicFaithQA Models answer short Islamic knowledge questions in Arabic and English. A model judges their answers against reference answers. | Qatar Computing Research Institute and collaborators | Expert-written questions | University | Not in cards | gpt-oss-20b 21.6% | Jan 2026 | Aug 2025 | ||
| LLM Round-Trip Translation Benchmark Models translate text into another language and back. The evaluation measures how well the original meaning survives. | Lech Mazur | Translation | University | Not in cards | GPT-5 8.69 | Sep 2025 | |||
| LLM SoccerArena Bundesliga 2026/27 Models forecast Bundesliga match scores and outcome probabilities with web access. The standings award points for score predictions made about 24 hours before kickoff. | LMU Munich, University of Cologne, and Paderborn University | Real-world events | Everyday | Not in cards | | May 2026 | |||
| LLM SoccerArena Premier League 2026/27 Models forecast Premier League match scores and outcome probabilities with web access. The standings use the predictions made about 24 hours before kickoff. | LMU Munich, University of Cologne, and Paderborn University | Real-world events | Everyday | Not in cards | GPT-5.5 89.0 | May 2026 | |||
| Long-Horizon Medical Agent Benchmark, Veterinary Clinic Ops v2 Agents operate a simulated veterinary clinic through its database API. Each of the 98 tasks checks whether the agent leaves the clinic records in the required state. | Praesidium Compliance Systems Corporation | Tasks from real jobs | Expert | Not in cards | GPT-5.6 Sol 100% | Sep 2026 | Sep 2026 | ||
| MANDOS Session 2 Models make process safety decisions across four private industrial scenarios. A model judge assesses each run. The leaderboard reports the share of passing runs. | RelayForge | Tasks from real jobs | Expert | Not in cards | Claude Sonnet 4.6 96.9% | Sep 2026 | Apr 2026 | ||
| MANDOS Run 003 Replaced in Sep 2026 | Not in cards | GPT-4o 100% | Jun 2026 | Mar 2025 | |||||
| MarineEVT Models answer 2,000 held-out questions about underwater events, including species interactions, movement, and timing. The authors report accuracy across five reasoning categories. | Hong Kong University of Science and Technology and collaborators | Video understanding | University | Not in cards | | Jun 2026 | Dec 2025 | ||
| McStasBench Agents design neutron scattering instruments. A simulator checks whether their designs reproduce published measurements. The authors score two additional instruments separately. | Oak Ridge National Laboratory | Lab and data work | Expert | Not in cards | Claude Sonnet 5 43.8% | Sep 2026 | Jul 2026 | ||
| MDMR-Bench Models generate one image from several references. A model judge checks whether the output preserves requested elements, follows instructions, and combines them correctly. | Bowen Zheng and colleagues | Image generation and editing | Everyday | Not in cards | GPT-Image-2 88.8% | May 2026 | Feb 2026 | ||
| MM-AlignBench Models answer open-ended questions about 252 human-annotated images. GPT-4o compares their answers with Claude-Sonnet-3 reference answers to assess alignment with human preferences. | Shanghai Jiao Tong University, Shanghai AI Laboratory, Nanjing University and others | General visual questions | Everyday | Not in cards | GPT-4V 82.5% | Feb 2025 | Nov 2024 | ||
| NARRA-Gym Agents build interactive stories around a user’s experiences and adapt scenes as the conversation unfolds. Judges rate story quality and user experience separately. | Yue Huang and colleagues | Creative writing | Everyday | Not in cards | Claude Sonnet 4.6 2 | May 2026 | Mar 2026 | ||
| NaturalBench Models answer 10,000 human-checked questions on natural images. Each question pairs with two images that need different answers, so a model cannot pass without looking. | Carnegie Mellon University and others (Baiqi Li, Zhiqiu Lin et al.) | General visual questions | Everyday | Not in cards | o3 55.1% | Oct 2024 | Apr 2025 | ||
| NoAdivines: Safe Order-Taking in Informal Spanish Models convert informal Latin American Spanish orders into structured actions. The grader checks whether they infer unsupported details or request clarification. | Henry Jeovany Rodezno | Chat and instructions in many languages | Everyday | Not in cards | gpt-oss-20b 100.0 | Oct 2026 | Sep 2026 | ||
| Numerical results extraction from randomized controlled trials Models extract group sizes, event counts, means, and standard deviations from trial reports. The test contains 656 treatment comparisons across 110 papers. | Northeastern University and King's College London | Medicine | University | Not in cards | | May 2024 | Sep 2026 | ||
| NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. | New York University (NYU Tandon) | Capture-the-flag challenges | University | Not in cards | Claude 3.5 Sonnet 22% | Jun 2024 | Apr 2025 | ||
| OASIS (agent safety) Agents handle multi-turn requests with different levels of complexity. Evaluators measure the proportion of safe responses. | OpenCompass and collaborators | Deception and alignment | University | Not in cards | GPT-5 mini 95.2% | Nov 2025 | Sep 2025 | ||
| OdysseyBench Agents recover requirements from conversation histories and complete workflows across documents, spreadsheets, PDFs, email, and calendars. Task-specific checks inspect the resulting file system and report the percentage of completed tasks. | Microsoft and University of Edinburgh | Documents and spreadsheets | University | Not in cards | o3 58.7% | Aug 2025 | Aug 2025 | ||
| OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. | Peking University, Alibaba Group, and Tsinghua University | Embodied reasoning | Everyday | Not in cards | | Sep 2026 | Sep 2026 | ||
| OpenComputer Computer-use agents complete 1,000 desktop tasks across 33 real applications, including browsers, office tools, creative software, development environments and messaging apps. App-specific verifiers inspect the final application state, and the harness gives partial credit from the recorded trajectory. | Yale NLP Lab | Desktop apps | Everyday | Not in cards | GPT-5.4 68.3% | May 2026 | Apr 2026 | ||
| OpenFrontiers-Bench Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings. | University of Illinois Urbana-Champaign (Tong Zhang) | Formal proofs | Research frontier | Not in cards | GPT-6.1 Sol 42.2% | Oct 2026 | Sep 2026 | ||
| PARCEL Models decide whether a New York court opinion supports, contradicts, or fails to address a cited claim. The benchmark contains 3,396 labeled claims. | University at Albany, SUNY | Law | University | Not in cards | | Oct 2026 | Dec 2025 | ||
| PhysClosureBench Agents run 50 expert-verified physics research tasks in strong-field and attosecond physics, deciding how to generate and validate evidence. A 20-criterion score and a closure level from RC0 to RC4 grade them. | Yijia Mao et al. | Research science | Research frontier | Not in cards | Claude Opus 5 6.29 | Oct 2026 | Sep 2026 | ||
| Poker Arena Models play no-limit Texas Hold’em across repeated sessions. The benchmark scores tournament chips and nine strategic dimensions, including memory and bluffing. | Pratham Singla, Shivank Garg, and Vihan Singh | Multi-agent games | Everyday | Not in cards | Claude Opus 4.6 15730.0 | Jun 2026 | Mar 2026 | ||
| PPTGen-Bench Agents create slide decks from text requests. Tencent renders the decks to PDF with LibreOffice. A judging agent assesses appearance and applies penalties for serious faults. | Tencent | Documents and spreadsheets | School | Not in cards | Claude Opus 5.5 67.7 | Nov 2026 | Nov 2026 | ||
| ReactionBench Models rank a viewer’s reactions, match a reaction to one of four video clips, and explain which cue prompted it. | MIT, University of Illinois Urbana-Champaign, and collaborators | Video understanding | Everyday | Not in cards | GPT-5 0.51 | Oct 2026 | Aug 2026 | ||
| Read-aloud Assessment Benchmark Speech models transcribe children reading aloud in Kiswahili. The evaluation measures transcription errors and disagreements with human assessors about which items the child misread. | Fab AI | Speech recognition and translation | Everyday | Not in cards | | Aug 2026 | Sep 2026 | ||
| ReplicationBench Agents reproduce computational results from published astrophysics papers. Evaluators check their numerical answers and assess whether their methods follow the original work. | Christine Ye and colleagues | Research science | Research frontier | Not in cards | Claude Sonnet 4.5 22% | Oct 2025 | Sep 2025 | ||
| RetailBench Agents operate a simulated supermarket for up to 180 days. They choose suppliers, order stock, and set prices as goods age and customers respond. | Ant Group and City University of Hong Kong | Running a business | University | Not in cards | GPT-5.5 $24,351 | Mar 2026 | Apr 2026 | ||
| RExBench Agents extend 12 published AI research projects from expert instructions. The evaluator runs their code against task-specific success criteria. Harbor distributes two public tasks. | Nicholas Edwards and colleagues | AI research tasks | Research frontier | Not in cards | Claude Opus 4.5 50% | Jun 2025 | Nov 2025 | ||
| RiOSWorld Computer-use agents work in an Ubuntu virtual machine through risky scenarios such as phishing emails and pop-ups. Scores track whether agents notice and avoid the risk. | RiOSWorld authors (Yang et al.) | Refusing harmful requests | Everyday | Not in cards | | May 2025 | Jun 2025 | ||
| RoughCut-Bench Models select and order video clips or mark speech to remove from a transcript. The evaluation compares cut plans with professional editors’ annotations. | Zhejiang University, ByteDance, and Peking University | Video understanding | University | Not in cards | | Jul 2026 | Jun 2026 | ||
| RuC-tt07-32k Models fill missing Verilog statements in Tiny Tapeout designs. A parser checks syntax, and a formal checker tests whether the completed design preserves behavior. | Barcelona Supercomputing Center | Electronics | University | Not in cards | | Apr 2026 | Sep 2025 | ||
| SAVU-Bench Models answer spatial questions about real scenes using sound and video. The authors report accuracy across seven tasks. | Yu Chen and colleagues | Audio understanding and voice | Everyday | Not in cards | | Oct 2026 | Jul 2026 | ||
| scBench (original study) Agents complete 394 tasks in single-cell data analysis. The authors compare their answers with results from the original analyses. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 4.6 52.8% | Feb 2026 | Feb 2026 | ||
| SciTaste-Bench Models judge the research value of papers in six disciplines. The benchmark compares their judgments with a reference index built from seven dimensions. | University of Chinese Academy of Sciences and collaborators | Research science | Expert | Not in cards | | Oct 2026 | Aug 2026 | ||
| SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. | Anthropic (with MATS and Anthropic Fellows) | Finding and exploiting real bugs | Expert | Not in cards | Claude Opus 4.5 $548,400,000 | Dec 2025 | Nov 2025 | ||
| SCUBA Agents work in a Salesforce sandbox as administrators, sales representatives, or service agents. Evaluators check milestones such as creating accounts and routing support cases. | Salesforce AI Research | Web browsing | University | Not in cards | GPT-5 53.9% | Sep 2025 | Aug 2025 | ||
| SemaPLC Function Track Agents generate individual Structured Text program units. A compiler and a formal verifier check the program against the requirements. Inconclusive checks count as failures. | Midea AIRC and collaborators | Electronics | Expert | Not in cards | GPT-5.5 82.1% | Aug 2026 | Jun 2026 | ||
| SemaPLC Project-Context Track Agents implement plant control logic inside existing Structured Text projects. The evaluator measures compilation, static checks, and agreement with reference outputs during execution. | Midea AIRC and collaborators | Electronics | Expert | Not in cards | GPT-5.5 100% | Aug 2026 | Jun 2026 | ||
| SF20K-Test Models answer questions about the story in a short film. The authors check multiple-choice answers against a key and grade free responses with GPT-3.5. | École Polytechnique and MBZUAI | Long videos | Everyday | Not in cards | LongVA-7B 62.8% | Jun 2024 | Sep 2024 | ||
| SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. | HCLTech Robotics | CAD models | Expert | Not in cards | GPT-6.1 Sol 100% | Oct 2026 | Nov 2026 | ||
| SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. | CMU | Genomics and life sciences | Expert | Not in cards | GPT-5.4 40.2% | May 2026 | Mar 2026 | ||
| SuperCLUE-AdsVU Models interpret video advertisements. | SuperCLUE | Video understanding | Everyday | Not in cards | | Jun 2026 | |||
| SuperCLUE-Agent Chinese language models are tested on agent skills in Chinese tasks. These include tool use with several API calls, task planning and memory over long dialogues. | CLUE benchmark team (SuperCLUE) | Function calling | Everyday | Not in cards | GPT-4o 78.4 | Oct 2023 | Jun 2024 | ||
| SuperCLUE-Edit Models edit an input image according to a Chinese instruction. The evaluation measures whether the requested change preserves relevant details. | SuperCLUE | Image generation and editing | Everyday | Not in cards | GPT-Image-2.5 Sunburst 82.5 | Oct 2025 | |||
| SuperCLUE-Fact Models answer short Chinese questions and identify false statements involving knowledge, common sense, misleading prompts and context. | SuperCLUE | Short factual questions | School | Not in cards | | Apr 2025 | Jun 2025 | ||
| SuperCLUE-I2V Models generate videos from still images. | SuperCLUE | Video generation | University | Not in cards | | ||||
| SuperCLUE-Role Models portray characters in Chinese conversations. A judging model rates aspects of the portrayal against reference answers. | SuperCLUE team | Chat and instructions in many languages | Everyday | Not in cards | GPT-4 Turbo (Jan 2024) 75.2 | Apr 2024 | Feb 2024 | ||
| SuperCLUE-Safety Chinese language models answer 4,912 open-ended safety questions over more than 20 sub-dimensions in multi-round adversarial dialogues. Follow-up turns push harder than the first prompt. | CLUE benchmark team (SuperCLUE) | Jailbreaks and prompt injection | Everyday | Not in cards | | Sep 2023 | Nov 2023 | ||
| SycoBench-600 Models decide whether to revise an answer after user feedback. The questions distinguish justified corrections from pressure to accept a false claim. | Debu Sinha | Wellbeing and sycophancy | Everyday | Not in cards | | Jul 2026 | Sep 2025 | ||
| SYNERGY systematic review screening Models rank titles and abstracts against a review’s eligibility criteria. Reviewers’ inclusion decisions supply the reference labels. | Utrecht University and collaborators | Medicine | University | Not in cards | Claude Opus 5.5 100% | Apr 2023 | Sep 2026 | ||
| Testing model's emotional inteligence Models identify the emotion in a story. A judge compares each response with the reference emotion on a sample of ten stories. | 23f3000941 | Reading comprehension | Everyday | Not in cards | Claude Haiku 4.5 100% | Oct 2026 | Sep 2026 | ||
| TRAJECT-Bench Agents select and execute tools to answer queries. The evaluation checks whether the actions satisfy a reference trajectory and produce the correct answer. | Pengfei He and collaborators, including Hippocratic AI | Function calling | University | Not in cards | | Aug 2025 | Sep 2025 | ||
| ts-bench Coding agents such as Claude Code, Codex and Gemini CLI solve TypeScript tasks, and tests decide pass or fail. Version 1 uses 25 Exercism practice exercises. Version 2 uses real tasks from a large TypeScript monorepo, taken from SWE-Lancer. | laiso (independent) | Function-level problems | University | Not in cards | Claude Sonnet 4.6 80% | Jun 2025 | Mar 2026 | ||
| TVRBench Agents move through simulated indoor scenes to reproduce a target image's viewpoint. Success requires stopping at exactly the target pose on the simulator's discrete grid. The evaluation split has 500 tasks across 80 scenes. | Zhejiang University | Simulated robots | Everyday | Not in cards | | May 2026 | Apr 2026 | ||
| Verify Gym Models give tighter rational bounds for roots on 120 held-out tasks. Exact integer arithmetic verifies both bounds and determines how much precision the answer adds. | carlos-toledo | Word problems | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| VeriGap Models implement programs and write tests from specifications. The harness checks program correctness, test validity, and whether the tests catch mutations. | Ahmed A. | Fixing real repositories | University | Not in cards | GPT-6.1 Sol 0.66 | Nov 2026 | Sep 2026 | ||
| VGI-Bench Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame. | University of Illinois Urbana-Champaign | Video world models | Everyday | Not in cards | | Aug 2026 | |||
| VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. | BUPT + Zhongguancun Academy + Peking University + Jilin University + Kling AI | Video generation | Expert | Not in cards | GPT-6 Astra 4.11 | Oct 2026 | Sep 2026 | ||
| VisualWebArena Multimodal web agents complete 910 visually grounded tasks on self-hosted classifieds, shopping and Reddit-style sites. Execution-based checks score each task. | Carnegie Mellon University | Web browsing | University | Not in cards | GPT-4V 16.4% | Jan 2024 | Apr 2024 | ||
| VLegal-Bench Models answer questions and solve legal scenarios under Vietnamese law. Legal experts check that reference answers follow authoritative documents. | Nguyen Tien Dong and colleagues | Law | University | Not in cards | | Dec 2025 | Sep 2025 | ||
| WildlifeBench Wildlife Knowledge V1 Models answer 185 questions about wild species without search or tools. The overall score weights ten tasks equally, including taxonomy, conservation status, and geographic range. | WildlifeBench | Science questions | University | Not in cards | | Nov 2026 | Oct 2026 | ||
| WISER-FindAll findall Entity discovery systems return all entities that match complex multi-criteria web queries. | Parallel Web Systems | Deep research | Expert | Not in cards | OpenAI deep research 21% | Nov 2025 | Feb 2025 | ||
| WISER-Search search Replaced in Nov 2025 | Not in cards | Jul 2025 | |||||||
| WISER-Atomic atomic Replaced in Jul 2025 | Not in cards | o3 69% | Apr 2025 | Sep 2025 | |||||
| xbench-Marketing Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers. | xbench (HongShan) | Running a business | Expert | Not in cards | o3 50.8 | Jun 2025 | May 2025 | ||
| 2026 Times Crossword Championship Models fill seven British cryptic crossword grids. The evaluator counts perfect grids and records the time and clues solved for each attempt. | The Times and Siddhartha Chatterjee | Puzzles and chess | Competition | Not in cards | GPT-6 Astra 100% | Oct 2026 | Sep 2026 | ||
| 3D SubwayBench Agents use Command Code to build a Subway Surfers style endless runner in a single HTML file. The scorecard grades six features from the submitted code. | Command Code | Building whole projects | University | Not in cards | Claude Opus 5 5.00 | Jul 2026 | Jul 2026 | ||
| 67AI Lab L2 nuclear medicine report structuring Models extract findings from 300 synthetic PET/CT and SPECT reports into a fixed form. They also flag conflicting values in 17 reports. | 67AI Lab | Medicine | University | Not in cards | | Nov 2026 | Sep 2026 | ||
| 67AI Lab L3 nuclear medicine guideline questions Models answer questions using eight retrieved passages from nuclear medicine guidelines. They must abstain when the passages lack an answer. | 67AI Lab | Medicine | University | Not in cards | Claude Opus 5.5 90.8% | Nov 2026 | Sep 2026 | ||
| AgentCLUE-Task Agents carry out interactive tasks in Chinese. | SuperCLUE | Apps and MCP servers | Everyday | Not in cards | Kimi K3 49.5 | Jul 2026 | |||
| AgenticRL-Bench Agents debug and maintain reinforcement learning infrastructure. A task passes when the implementation meets every hidden check. | The Environment Lab at Eigent AI | Machine learning engineering | Expert | Not in cards | GPT-6.1 Sol 22.8% | Nov 2026 | Nov 2026 | ||
| AgentKernelArena Coding agents optimize GPU kernels through full workflows with compilers and profilers. Scores check speedups and whether gains generalize beyond tuned shapes. | AMD | GPU kernels | Expert | Not in cards | Claude Opus 4.6 3.23× | May 2026 | Apr 2026 | ||
| AGILE Automatic Genre Identification Benchmark Models assign one of eight genres to web texts, including news, opinion, and advertising. The test covers English and ten other languages. | Taja Kuzman, Igor Mozetič, and Nikola Ljubešić | Reading in many languages | Everyday | Not in cards | GPT-5 mini 0.82 | Jun 2023 | Sep 2025 | ||
| AI GameStore Vision-language models play short episodes of 100 games adapted from Apple App Store and Steam top charts. Each game score is divided by the average human score, and models are ranked by the geometric mean. | Lance Ying, Joshua B. Tenenbaum and collaborators | Video games | Everyday | Not in cards | | Feb 2026 | Dec 2025 | ||
| AI Security Leaderboard Models face systematic jailbreak attacks across chemical, biological, radiological, nuclear, explosive and cyber domains. Scores count universal jailbreaks and estimate attacker cost. | FAR.AI | Jailbreaks and prompt injection | Expert | Not in cards | Grok 4.5 $60 | Jul 2026 | Sep 2026 | ||
| AMLBench v0.2 (Superjective) Models decide whether to escalate transaction alerts. The score measures how often they escalate reportable alerts under changes to analyst instructions and case narratives. | Scott Burns and Superjective | Finance | University | Not in cards | | Jul 2026 | May 2026 | ||
| AnalogCoder-Pro Agents write analog circuit programs and use simulated waveforms to diagnose failures. The authors report the fraction of successful designs and the number of tasks solved. | Yao Lai and colleagues | Electronics | Expert | Not in cards | Claude 3.7 Sonnet 88.4% | Sep 2025 | |||
| AnalogPortBench Models adapt analog circuit netlists to a different chip process or flag infeasible specifications. A simulator checks the designs against electrical limits. | University of Delaware | Electronics | Expert | Not in cards | GPT-6.1 Sol 56% | Oct 2026 | Oct 2026 | ||
| AstroVLBench Models interpret galaxy images, light curves, and spectra across five observing tasks. The authors compare their classifications with reference labels and examine their explanations. | Wenke Ren and colleagues | Lab and data work | University | Not in cards | | Apr 2026 | Feb 2026 | ||
| Atrex-Bench Agents implement GPU kernels for 30 operators and 440 shapes sampled from production inference workloads. The evaluator checks compilation and numerical correctness, then scores performance against hardware roofline ceilings with weights from production GPU usage. | Alibaba | GPU kernels | Expert | Not in cards | GPT-5.5 0.11 | Jun 2026 | May 2026 | ||
| AutoCAD Bench Computer-use agents turn a dimensioned reference drawing into a correct, editable DWG in AutoCAD through the screen alone, with mouse and keyboard. 50 tasks span 2D drafting and 3D modeling at four difficulty levels. | Markov | CAD models | Expert | Not in cards | GPT-5.6 Sol 46% | Jul 2026 | Jul 2026 | ||
| AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. | University of California, Santa Cruz and NVIDIA | Machine learning engineering | Expert | Not in cards | Claude Opus 4.6 66.5 | Jun 2026 | Mar 2026 | ||
| BABILong Models answer bAbI reasoning questions whose supporting facts are scattered through very long distractor text, at lengths up to millions of tokens. | AIRI and collaborators (Kuratov et al.) | Reasoning over long documents | Everyday | Not in cards | GPT-4 Turbo (Jan 2024) 72.4% | Jun 2024 | May 2024 | ||
| Behavior2Code Agents reconstruct command line programs from documentation and observed behavior. Hidden tests compare each implementation with the original. | Turing | Building whole projects | Expert | Not in cards | GPT-6 Astra 31.7% | Aug 2026 | Sep 2026 | ||
| BIBIMBAP Agents read and edit IFC building files. Executable tests compare their changes and answers with reference results. | University of Rostock and Clausthal University of Technology | CAD models | University | Not in cards | Claude Opus 4.6 50.2% | Mar 2026 | Mar 2026 | ||
| BLUR Models identify a remembered item from incomplete clues in several languages and media. | Patronus AI | Questions that need search | Everyday | Not in cards | OpenAI Computer-Using Agent (CUA) 54% | Mar 2025 | May 2025 | ||
| BrowseSafe-Adapt Browser agents face adaptive prompt injections across 1,024 tasks. The score is attack success within 64 attempts. | Perplexity | Jailbreaks and prompt injection | University | Not in cards | Claude Sonnet 5 30.7% | Oct 2026 | Jul 2026 | ||
| BuildArena 2.0 Construction Challenge Agents build and test machines in Besiege for seasonal engineering challenges. The organizers score each entry using simulation results and the rules for its track. | BuildArena team at Westlake University | Video games | Competition | Not in cards | Jun 2026 | ||||
| BuildArena 1.0 Replaced in Jun 2026 | Westlake University | University | Not in cards | GPT-5 45.3% | Oct 2025 | Aug 2025 | |||
| Cekura Text to Speech Round-trip word error rate averaged across the two transcribers, as displayed by Cekura. | Cekura | Audio understanding and voice | Everyday | Not in cards | Fish Audio Drama 3 1.92% | Jul 2026 | Nov 2026 | ||
| Chatbots and the Ballot Box Models answer voter questions in English and Spanish across ten US states. Analysts check routine answers against election guidance. A model grades responses to disputed claims. | Institute for Strategic Dialogue | Long-form factuality | Everyday | Not in cards | GPT-5.5 89.3% | Sep 2026 | May 2026 | ||
| CheckerBench Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback. | Humanlaya Data Lab | Fixing real repositories | Expert | Not in cards | Claude Opus 4.8 45.3% | Oct 2026 | Aug 2026 | ||
| ChJGBench Models draft the reasoning and orders in Chinese civil, criminal, and administrative judgments. The evaluator compares their work with 3,462 real judgments. | Zhejiang University and collaborators | Law | Expert | Not in cards | | Nov 2026 | Mar 2026 | ||
| CircuitSense Models read circuit schematics and block diagrams, derive equations, and select designs. SymPy checks symbolic answers. Other tasks use exact matches, a model judge, or simulation. | Northeastern University and Brookhaven National Laboratory | Visual math and diagrams | University | Not in cards | | Sep 2025 | Aug 2025 | ||
| Civil-Bench Models answer Chinese civil engineering licensing questions. Answer keys grade the objective questions. In the paper, GPT-4 Turbo grades written answers against individual scoring points. | Tongji University | Exam questions | University | Not in cards | | May 2025 | |||
| CLBench-V Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition. | CLBench-V authors | General visual questions | Everyday | Not in cards | | Jul 2026 | Apr 2026 | ||
| CMB Models answer 11,200 Chinese medical exam questions and questions about 74 clinical cases. Exam answers receive exact-match scores. A model judge rates clinical answers for fluency, relevance, completeness, and medical proficiency. | Freedom Intelligence | Medicine | Expert | Not in cards | | Aug 2023 | Nov 2023 | ||
| CNN/DailyMail Summarization Models summarize news articles from CNN and the Daily Mail. The harness compares generated summaries with the publishers’ highlights. | Karl Moritz Hermann and colleagues | Faithful to sources | School | Not in cards | | Jun 2015 | Jul 2022 | ||
| Cognition CodeSearch Eval Agents retrieve relevant files and line ranges for questions about real repositories. Cognition compares their answers with human labels using precision-weighted F scores. | Cognition | Fixing real repositories | University | Not in cards | GPT-5 0.59 | Oct 2025 | Oct 2025 | ||
| Command Code Flappy Bird Agents use Command Code to build a Flappy Bird game with Canvas. The scorecard grades features including collisions, window resizing, and offline play. | Command Code | Building whole projects | University | Not in cards | GLM-5.2 4.83 | Jul 2026 | Jul 2026 | ||
| Command Code Pixel Art Space Shooter Agents use Command Code to build a scrolling Canvas game with cave walls and collectible gems. The scorecard grades features including collisions and offline play. | Command Code | Building whole projects | University | Not in cards | Grok 4.5 4.83 | Jul 2026 | Jul 2026 | ||
| Competitive Programming Code Generation Models solve 120 Codeforces problems in Python. The harness runs their submissions against test cases. | Montasir | Competitive programming | Competition | Not in cards | GPT-6.1 Sol 90% | Oct 2026 | Sep 2026 | ||
| ConceptEdit-Bench Models edit images according to 1,000 instructions across six categories. A vision-language judge scores instruction following, visual quality, and consistency in unedited regions from 0 to 10. The per-case score averages those three dimensions. | Shanghai Jiao Tong University and Ant Group | Image generation and editing | Everyday | Not in cards | | Aug 2026 | Feb 2026 | ||
| CooperBench Two coding agents each get a different feature for the same repository and must coordinate so both changes work together. Over 600 tasks span 12 libraries in 4 languages. | Stanford NLP | Fixing real repositories | Expert | Not in cards | GPT-5 27.9% | Jan 2026 | Dec 2025 | ||
| CTF-Archive Diamond Agents solve 285 difficult capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. | US CAISI (NIST) | Capture-the-flag challenges | Expert | Not in cards | GPT-5.5 71% | May 2026 | Jun 2026 | ||
| CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. | Vals AI | Capture-the-flag challenges | Expert | Not in cards | Claude Opus 5.5 87.7% | Oct 2026 | Oct 2026 | ||
| CyberStrike Agents complete cybersecurity tasks. Automated checks grade the outcomes. | Turing | Secure code and review | Expert | Not in cards | GPT-5.6 Sol 31.5% | Aug 2026 | Jul 2026 | ||
| DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. | ByteDance Seed | Data science and SQL | Expert | Not in cards | GPT-5 47% | Dec 2025 | Aug 2025 | ||
| DataPrep-Bench Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score. | Peking University | Machine learning engineering | Expert | Not in cards | | May 2026 | Feb 2026 | ||
| Dấu or Die: Vietnamese Diacritic Restoration Models restore missing diacritics in 37 Vietnamese sentences. A sentence earns credit only when every mark matches the reference. | tuan nguyen | Reading in many languages | School | Not in cards | | Oct 2026 | Oct 2026 | ||
| DEEPSYNTH Agents search multiple websites and combine evidence to answer 120 research tasks across seven domains. They return structured JSON answers scored with precision, recall, F1, and exact match. A separate model judge assesses answers beyond exact matching. | Huawei Noah's Ark Lab and university collaborators | Deep research | Expert | Not in cards | OpenAI deep research 8.97% | Feb 2026 | Dec 2025 | ||
| Document AI Benchmark Arena Models extract fields from business documents using a common schema and OCR text. The scorer normalizes numbers and dates and allows small spelling differences in names and addresses. | DocAI Fabric | Documents, charts and text in images | Everyday | Not in cards | GPT-4.1 92.6% | Feb 2026 | Jun 2025 | ||
| E/RP Leaderboard by PsychoSmiley Models answer 25 yes or no questions and write three roleplay responses. Claude Sonnet 4.5 grades the two parts for up to 325 points. | PsychoSmiley | Creative writing | Everyday | Not in cards | Kimi K2.5 289.0 | Nov 2025 | Jun 2026 | ||
| EgoProceVQA Models watch first-person clips and answer questions about the steps in a procedure. Some questions ask when a step occurred or which step is missing. | The Hong Kong Polytechnic University and South China University of Technology | Video understanding | Everyday | Not in cards | GPT-5.1 51.6% | Jul 2026 | Dec 2025 | ||
| EKWbench Agents complete operational tasks such as sourcing reviews and referral triage. The score averages the share of verifier weight earned. | Turing | Tasks from real jobs | Expert | Not in cards | Claude Opus 5 74.4% | Jul 2026 | |||
| EmbodiedCLUE-World 2.0 World models predict outcomes in embodied settings under the second evaluation protocol. | SuperCLUE | Interactive world models | Everyday | Not in cards | | Jul 2026 | Jul 2026 | ||
| EmbodiedCLUE-World Original Replaced in Jul 2026 | Not in cards | | May 2026 | ||||||
| ExcelFixBench Agents repair formulas, data, and formatting in an existing workbook. The task rubric checks the repair and whether unrelated content survived. | Tencent | Documents and spreadsheets | University | Not in cards | | Nov 2026 | Nov 2026 | ||
| Extract Bench (Contextual AI) Models extract structured JSON from PDFs. Field checks include exact matches, numerical tolerances and semantic judgments. | Contextual AI | Documents, charts and text in images | University | Not in cards | | Feb 2026 | Dec 2025 | ||
| Factorio Learning Environment Agents write Python programs that control a Factorio factory, in lab tasks with fixed resources and in open play. Scored by production throughput, milestones and lab task success. | Jack Hopkins et al. (FLE team) | Video games | Expert | Not in cards | Claude 3.5 Sonnet (October 2024) 293206.0 | Mar 2025 | Feb 2025 | ||
| Faroese Grammatical Correctness Models choose the correct Faroese sentence from a pair drawn from student essays and their corrections. | University of the Faroe Islands and University of Copenhagen | Reading in many languages | Everyday | Not in cards | GPT-5 90.9% | May 2026 | Aug 2025 | ||
| FinBen A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading. | The Fin AI (Qianqian Xie and coauthors) | Finance | University | Not in cards | ChatGLM3-6B 0.25 | Feb 2024 | Apr 2024 | ||
| FinStepBench CorporateValuation Models value companies using financial evidence and calculations. The score is the published CorporateValuation percentage. | StepFun | Finance | Expert | Not in cards | GPT-6 Astra 77.3% | Sep 2026 | Sep 2026 | ||
| FinStepBench DeepResearch Models write finance research reports. The score is the published DeepResearch percentage. | StepFun | Finance | Expert | Not in cards | Claude Opus 5 59.1% | Sep 2026 | Sep 2026 | ||
| FinStepBench LiveSearch Models retrieve and verify live financial facts. The score is the published LiveSearch percentage. | StepFun | Finance | Expert | Not in cards | | Sep 2026 | Sep 2026 | ||
| FraudBench (GOAL Lab) Models face 159 scenarios drawn from 112 real U.S. fraud lawsuits. Yes or no questions, open questions and agent tasks test whether a model recommends, assists or commits fraud. | GOAL Lab | Refusing harmful requests | University | Not in cards | gpt-oss-120b 10.1% | Nov 2026 | Nov 2025 | ||
| GarbleBench Models answer factual questions after controlled keyboard typos, dropped letters, stray characters and swaps corrupt the text. | Adam Holter | Instructions and chat | Everyday | Not in cards | | Oct 2026 | Oct 2026 | ||
| GenomeQA Models answer multiple-choice questions about short genomic sequences. The benchmark covers six task families, including species classification. | Weicai Long and colleagues | Genomics and life sciences | University | Not in cards | | Apr 2026 | Nov 2025 | ||
| GTO Wizard Benchmark Models play heads-up no-limit Texas Hold’em against GTO Wizard AI. AIVAT reduces variance while the benchmark reports poker winnings. | GTO Wizard research team | Multi-agent games | Expert | Not in cards | GPT-5.5 -9.23 | Mar 2026 | Jul 2026 | ||
| GUI-CEval Models answer questions about Chinese app screens, locate controls, and choose actions. Agents also complete tasks on physical devices, where the grader checks interface elements. | Xiaomi HyperAI Team | Phones | Everyday | Not in cards | | Mar 2026 | May 2025 | ||
| HABIT-Bench Agent memory systems must infer user habits from weak evidence spread across many past conversations and apply them correctly later. | Shanghai Jiao Tong University (LUMIA) | Learning from the context | Everyday | Not in cards | GLM-5.3-Flash 42% | Oct 2026 | Aug 2026 | ||
| HarnessDev Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score. | ByteDance Seed | AI research tasks | Research frontier | Not in cards | Claude Opus 4.8 67.8% | Sep 2026 | May 2026 | ||
| HLE-Verified (text-only Full) Models answer the text-only questions of Humanity's Last Exam after expert verification and revision. The Full set keeps the questions experts marked uncertain, and the score is pass@1 accuracy. | SKYLENAGE AI | Expert-written questions | Expert | Not in cards | Claude Opus 4.6 46.8% | Feb 2026 | Feb 2026 | ||
| HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. | Ramy ElMallah and colleagues | Simulated robots | Everyday | Not in cards | GPT-6 Astra 61.9% | Nov 2026 | Sep 2026 | ||
| ICML 2025 Experts Benchmark Models answer tasks submitted by participants at ICML 2025 in Vancouver. Kaggle reports overall performance and individual task outcomes. | Kaggle | Language reasoning | University | Not in cards | | Jul 2025 | Jun 2025 | ||
| Infrathon Agents optimize AI infrastructure under correctness and fidelity constraints. Hidden workloads measure performance gains, with additional checks for rule compliance. | University of Washington, Ai2, and collaborators | Speed and optimization | Expert | Not in cards | Claude Opus 5.5 91.2 | Nov 2026 | Sep 2026 | ||
| InteractiveBench Logic A model asks restricted questions to recover the hidden explanation of a situation puzzle. It solves an item only if a judge accepts its final explanation within 20 rounds. | InteractiveBench and Princeton University | Abstract puzzles and logic | Everyday | Not in cards | | Mar 2026 | Dec 2025 | ||
| InteractiveBench Math A model solves mathematical problems while querying a judge about intermediate claims. The judge holds a reference derivation and marks final solutions correct or incorrect under a fixed interaction budget. | InteractiveBench and Princeton University | Research mathematics | Expert | Not in cards | Grok 4.1 Fast 76.9% | Mar 2026 | Dec 2025 | ||
| InternBootcamp Models solve generated reasoning tasks with task-specific answer checkers. The hub port lets users select a task and sample examples with a fixed seed. | InternLM | Language reasoning | University | Not in cards | | May 2025 | May 2025 | ||
| iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. | Carnegie Mellon University (Salakhutdinov lab) | Phones | Everyday | Not in cards | Claude Opus 4.6 51.9% | Jun 2026 | Mar 2026 | ||
| JevBench Decision models, such as TypeSafe AI's Jev and compatible systems, answer typed decision questions. The board scores accuracy, probability calibration, latency and cost. | Benchmark Heaven | Other | Everyday | Not in cards | GPT-6 Luna 98.4% | Sep 2026 | Sep 2026 | ||
| JFBench Models answer Japanese prompts with multiple verifiable constraints. A response passes only if it satisfies every constraint. | Preferred Networks | Chat and instructions in many languages | Everyday | Not in cards | | Mar 2026 | Aug 2026 | ||
| JusticeAxis Models assess evidence and produce judgments for 256 criminal cases from 18 countries. The evaluator checks charges and outcomes against lawyer-written references. | The Chinese University of Hong Kong and collaborators | Law | Expert | Not in cards | | Sep 2026 | Sep 2026 | ||
| K12-Bench Models select curriculum concepts, prerequisite relationships, related knowledge, experimental evidence, and textbook locations. The 23,640 multi-select questions come from a textbook knowledge graph. Exact match and instance-level F1 compare selected options with graph-derived answers. | Peking University, Institute for Advanced Algorithms Research Shanghai, OriginHub Technology, and Zhongguancun Academy | Language reasoning | School | Not in cards | | May 2026 | Apr 2026 | ||
| K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. | Chong Li and colleagues | Other professions | School | Not in cards | | Jun 2025 | Jan 2025 | ||
| K9-Bench Models answer 4,744 questions about dog behavior in 907 videos. The evaluator compares each free-form answer with five options that the model never sees. | Ogmen Robotics and Georgia Institute of Technology | Video understanding | School | Not in cards | | Jun 2026 | Oct 2025 | ||
| Kaggle's Task Tuesday Benchmark Models attempt tasks selected by the Kaggle team from community submissions. The board reports an overall score and individual task results. | Kaggle | Language reasoning | University | Not in cards | | Jan 2026 | Dec 2025 | ||
| KnowU-Bench Agents use Android apps such as Mattermost and Taodian. They infer preferences, ask for missing details, and decide when to act or stay silent. | Zhejiang University, Apple, and Tencent | Phones | Everyday | Not in cards | Claude Sonnet 4.6 60.4% | Apr 2026 | Feb 2026 | ||
| Legacy-Bench Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass. | Factory | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 60% | Sep 2026 | Sep 2026 | ||
| LiveSearch Agents search for current financial figures. The evaluator runs a stored query against a professional data service to update the reference answer. | Sinuo Wang and colleagues | Finance | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| LLM Fantasy Football Arena Guillotine Models draft NFL players and bid for free agents. Each week the lowest-scoring team leaves, and the remaining teams bid on its players. | Wildcard Labs | Multi-agent games | Everyday | Not in cards | | Sep 2026 | Aug 2026 | ||
| LLM SoccerArena La Liga 2026/27 Models forecast La Liga match scores and outcome probabilities with web access. The standings award points for score predictions made about 24 hours before kickoff. | LMU Munich, University of Cologne, and Paderborn University | Real-world events | Everyday | Not in cards | Claude Opus 4.8 129.0 | May 2026 | |||
| LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. | Songhan Zuo and colleagues | Law | Expert | Not in cards | Kimi K2.5 0.66 | Jun 2026 | Apr 2026 | ||
| LoopArena Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit. | AMAP-ML (Alibaba) | Building whole projects | Expert | Not in cards | GPT-5.5 36.3% | Aug 2026 | Jul 2026 | ||
| LT-History Models answer questions in Lithuanian about national and world history. | Yevhen Kostiuk and colleagues | Knowledge in many languages | School | Not in cards | | Mar 2025 | Sep 2024 | ||
| Math24o Models solve Chinese high school mathematics olympiad preliminary problems from 2024. A program compares their final answers with reference answers. | SuperCLUE team | Competition problems with answers | Competition | Not in cards | o3-mini 85.7% | Mar 2025 | Mar 2025 | ||
| MatSciBench Models solve college materials science problems, including questions with figures. The authors report accuracy separately for text and image questions. | Junkai Zhang and colleagues | Science questions | University | Not in cards | GPT-5 64% | Oct 2025 | Aug 2025 | ||
| Med-HALT Models answer medical exam questions designed to reveal hallucinations. The hub port uses the false-confidence, fake-question, and none-of-the-above reasoning tasks. | Med-HALT authors | Hallucination rates | Expert | Not in cards | | Jul 2023 | Jul 2023 | ||
| Mind the Gap Models describe visible events during natural pauses in a video's soundtrack. The evaluator renders the narration and checks its timing and coverage. | Google Research and UC Berkeley | Video captioning | University | Not in cards | | Oct 2026 | May 2026 | ||
| MIRAGE-MMMT Models decide whether to ask for more information or answer a grower during an agricultural consultation. They then write the question or answer. | University of Illinois Urbana-Champaign and Amazon | Multi-turn conversation | University | Not in cards | GPT-4o 65.5% | Jun 2025 | Apr 2025 | ||
| MooseBench Models write input files for MOOSE, a multiphysics simulator. The grader reconstructs each file’s governing equations and compares them with the intended physics. | Zhenghan Song, Yulong Liu, and colleagues | Scientific code | Expert | Not in cards | Claude Sonnet 4.6 0.85 | May 2026 | Apr 2026 | ||
| Mosaic Eval Gauntlet v0.3.0 Models complete tasks across reading, reasoning, world knowledge, language understanding, and long context. MosaicML groups results by capability. | MosaicML | Language reasoning | University | Not in cards | | Sep 2023 | |||
| NAFD E1 Financial Reporting Models sit professional finance exams on topics including financial reporting, planning, and taxation. A grader compares their answers with the official keys. | Nelson VIKOU | Finance | University | Not in cards | Claude Opus 5 0.93 | Nov 2026 | Sep 2026 | ||
| NatureBench Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results. | Frontis AI | Research science | Research frontier | Not in cards | Claude Opus 5 23.3% | Jun 2026 | Jul 2026 | ||
| NautData test set Models answer questions, describe underwater scenes, and locate organisms in images. The paper reports 7,916 test examples on 3,920 images. | Huazhong University of Science and Technology and National University of Defense Technology | General visual questions | University | Not in cards | | Oct 2025 | Feb 2025 | ||
| NewsBench (Chinese journalism) Models complete Chinese newsroom assignments and answer multiple choice questions about writing and editorial safety. GPT-4 evaluates the written responses. | Miao Li and colleagues at the Institute for Advanced Algorithms Research, Shanghai, and partner institutions | Other professions | University | Not in cards | GPT-4 Turbo 2.44 | Feb 2024 | Nov 2023 | ||
| norregel Models supply the numeric value of a Norwegian legal provision on a specified date. The evaluator distinguishes current values from superseded ones. | Eirik Botten Nicolaysen | Law | University | Not in cards | Claude Opus 4.5 45.1% | Oct 2026 | Apr 2026 | ||
| Norwegian Courts Bitext Mining Models match court sentences written in Norwegian Bokmål and Nynorsk. The test contains 228 aligned pairs and scores matches with F1. | Scandinavian Embedding Benchmark team | Translation | School | Not in cards | | Jan 2024 | Oct 2026 | ||
| OmniEarth-Bench Models interpret satellite images and observational data across Earth systems, answering questions about environmental processes and their interactions. Multiple-choice accuracy and model judgments score answers. Separate measures check localization, captions, and annotated reasoning steps. | Shanghai Artificial Intelligence Laboratory and university collaborators | Research science | Expert | Not in cards | | May 2025 | Mar 2025 | ||
| OpenBounds V2 Pilot Agents schedule jobs under machine and resource constraints. The evaluation compares each schedule with a greedy baseline and a reference solution. | Andre Albrecht | Speed and optimization | University | Not in cards | | Nov 2026 | Sep 2026 | ||
| OpenStudio-MCP LLM Test Benchmark v1.2.1 Agents change building energy models with OpenStudio tools. The grader checks the tool calls and inspects the resulting model against task requirements. | National Laboratory of the Rockies | Apps and MCP servers | University | Not in cards | Claude Opus 4.6 100% | Aug 2026 | May 2026 | ||
| OpenThoughts-TBLite Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates. | OpenThoughts Agent team, Snorkel AI and Bespoke Labs | Terminal tasks | Expert | Not in cards | Kimi K2.5 75.1% | Feb 2026 | Jan 2026 | ||
| OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. | Shanghai AI Laboratory (OpenRobotLab) and collaborators | Embodied reasoning | Everyday | Not in cards | GPT-4.1 53.4% | Jul 2025 | Apr 2025 | ||
| PCA-Bench Models interpret visual scenes, explain their reasoning, and select actions for embodied tasks. GPT-4-0125 compares perception concepts, reasoning, and actions with annotated references. | Peking University and Alibaba Group | Embodied reasoning | Everyday | Not in cards | Qwen-VL-Max 0.70 | Feb 2024 | Mar 2024 | ||
| PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. | Positronic Robotics | Real robots | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| PhoneShell iOS Agent Benchmark Agents operate apps on a physical iPhone to set timers, find books, and complete other everyday tasks. Automated checks read the resulting device state. | Blolabel | Phones | Everyday | Not in cards | Kimi K3 96.6% | Sep 2026 | Jul 2026 | ||
| Pieverse Agent Prediction Arena Agents trade event contracts on Polymarket with $1,000 each in real money. Pieverse tracks account value as market prices change. | Pieverse | Real-world events | University | Not in cards | Claude Sonnet 4.5 $981 | Dec 2025 | Dec 2025 | ||
| Pigeonholing Partisans four-word generation Models use a survey respondent’s profile to generate four words describing supporters of each major US party. One measure counts how often raters identify the writer’s party. | Lisa P. Argyle and colleagues | Social simulation | Everyday | Not in cards | | Sep 2022 | Jun 2024 | ||
| Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. | Snorkel AI | Deception and alignment | Everyday | Not in cards | Claude Sonnet 5.5 0.59% | Sep 2026 | |||
| PLM-VideoBench Models answer questions about activities, describe events involving a particular person or object, and locate those events in time. Meta also tests first-person video understanding. | Meta FAIR | Video captioning | Everyday | Not in cards | | Apr 2025 | Feb 2025 | ||
| Precedent vs. Spec Models complete tables whose example rows conflict with the written rule. The grader checks which rule they followed and whether they mention the conflict. | Finn Mead | Following precise instructions | School | Not in cards | GPT-5.5 62.5% | Oct 2026 | Aug 2026 | ||
| Pressure-Tested Alignment Models choose how to act in scenarios where termination threatens their objective. A judge checks whether they take a shortcut that games the reward. | Ram Acharya | Deception and alignment | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| ProjectTest Models write unit tests for existing Python, Java, and JavaScript projects. The authors run the tests and measure code coverage. | Yibo Wang and colleagues with Scale AI | Fixing real repositories | University | Not in cards | | Feb 2025 | Nov 2025 | ||
| ProOPF-B Models translate operational requirements into executable optimal power flow programs. They change grid parameters, objectives, and constraints. The evaluator checks implementations against expert references. | Zhejiang University, Peking University, Xi’an Jiaotong University, and China Southern Power Grid | Scientific code | Expert | Not in cards | | Feb 2026 | Dec 2025 | ||
| Putnam-like Models write proofs for 96 university problems in 8 sets of 12, in the style of the easier and middle Putnam tiers. Each problem has a Lean statement and a point rubric for grading. | Google DeepMind | Written proofs | University | Not in cards | | Sep 2025 | Jun 2025 | ||
| PuzzleWorld Models solve 667 puzzle-hunt problems that require varied reasoning strategies. | PuzzleWorld authors | Puzzles and chess | Competition | Not in cards | | Jun 2025 | Nov 2025 | ||
| Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. | Ramp Labs | Finance | Expert | Not in cards | Claude Opus 4.6 62.1% | Jun 2026 | May 2026 | ||
| RECODE-H Agents implement research code from papers and repositories. They can request feedback from a simulated collaborator before their code runs against tests. | Chunyu Miao and colleagues | AI research tasks | Expert | Not in cards | GPT-5 84.3% | Oct 2025 | Sep 2025 | ||
| RiceChem Models grade long answers to four college chemistry questions. Each of the 1,264 answers has human judgments for the applicable rubric items. | Shashank Sonkar and colleagues at Rice University | Other professions | University | Not in cards | | Apr 2024 | Apr 2024 | ||
| RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. | Fudan University, Shanghai Innovation Institute, and Shanghai AI Laboratory with collaborators | Puzzles and chess | Everyday | Not in cards | | Jun 2026 | Jun 2026 | ||
| RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. | Changxuan Fan and colleagues | Simulated robots | University | Not in cards | GPT-6.1 Sol 80.6% | Nov 2026 | Oct 2026 | ||
| ROSTrans Models migrate robot software from ROS 1 to ROS 2. The evaluator builds each package, runs its nodes, and tests their behavior. | New York University | Fixing real repositories | University | Not in cards | Claude Opus 4.6 10.9% | Oct 2026 | Apr 2026 | ||
| RSI Control Arena An agent does one of 14 tasks from a successor model's training pipeline, such as SFT or reward modeling, while covertly trying to install one of 16 misaligned properties in that successor. A held-out oracle checks whether the property landed, and LLM monitors try to flag the attack from the transcript. | Scale AI | Deception and alignment | Research frontier | Not in cards | | Oct 2026 | Sep 2026 | ||
| rTaste Agents propose, rank, and forecast experiments inspired by recent research papers. An independent executor attempts the experiments, and auditors assess their results. | University of Minnesota and collaborators | AI research tasks | Expert | Not in cards | GPT-6.1 Sol 3.71 | Oct 2026 | Sep 2026 | ||
| Safety-Scene Vision Benchmark Models identify visible hazards in everyday photos. The evaluation compares answers under a neutral prompt and a prompt claiming an inspector found danger. | Clivin John | General visual questions | Everyday | Not in cards | | Oct 2026 | Aug 2026 | ||
| ScienceAgentBench Agents write programs for 102 data-driven discovery tasks from 44 papers in four disciplines. Outputs are checked by execution and task-specific metrics. | Ohio State University (OSU NLP Group) | Research science | Expert | Not in cards | o1-preview 42.2% | Oct 2024 | Sep 2024 | ||
| SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. | C5R | Lab and data work | Expert | Not in cards | Claude Fable 5.1 45.3% | Sep 2026 | Sep 2026 | ||
| SDR-Bench Models personalize sales pitches using customer research. A judge compares each pitch with points from successful human outreach. | Ashutosh Srivastava and colleagues | Running a business | University | Not in cards | Claude Sonnet 4.6 55.8% | Feb 2026 | Mar 2026 | ||
| Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. | Columbia University, Stanford University, and collaborators | Simulated robots | Everyday | Not in cards | Grok 4.3 42.9% | Nov 2026 | Aug 2026 | ||
| ShadowBench Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match. | Perpetual Research | Spatial reasoning and counting | Everyday | Not in cards | Claude Opus 5.5 48.8 | Nov 2026 | Oct 2026 | ||
| SilverquiLLM-bench hob-medium Coding agents implement five Hobbit cards in a Python game engine. The grader checks card behavior with 90 hidden tests. | Snowfox Builds | Fixing real repositories | University | Not in cards | GPT-6.1 Sol 96.9% | Oct 2026 | Sep 2026 | ||
| SimuVerity Agents build executable Simulink models from text for 101 tasks in ten engineering domains. Simulations check that each model meets its engineering requirements. | Xi'an Jiaotong University | Electronics | Expert | Not in cards | Claude Opus 4.8 42.9 | Oct 2026 | Aug 2026 | ||
| Solana Machine Trading Agents write trading bots for hidden market scenarios replayed on offline Solana snapshots. The headline score compares total profit and loss against a reference bot. | Solana Foundation | Finance | Expert | Not in cards | GPT-6 Astra 6.11 | Oct 2026 | Nov 2026 | ||
| SpatialBench (original study) Agents complete 146 tasks in spatial transcriptomics. The authors compare their answers with results from the original analyses. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 4.5 38.4% | Dec 2025 | Dec 2025 | ||
| STATE-Bench 0.8.0 Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. | Microsoft | Customer service | Expert | Not in cards | GPT-5.5 58.9% | Jun 2026 | Apr 2026 | ||
| STATE-Bench Main 0.7.0 Replaced in Jun 2026 | Not in cards | GPT-5.4 55.7% | May 2026 | Apr 2026 | |||||
| Supabase Evals Agents build, deploy, investigate, and repair Supabase projects. The hub port uses OpenCode and runs the original task scorers from a pinned release. | Supabase | Apps and MCP servers | Expert | Not in cards | Claude Opus 5.5 92.8% | Apr 2026 | Sep 2026 | ||
| SuperCLUE-Industry Models answer multi-turn questions about industrial work in Chinese. A judging model applies task-specific criteria. | SuperCLUE team | Tasks from real jobs | University | Not in cards | GPT-4 Turbo (Jan 2024) 74.8 | Mar 2024 | Feb 2024 | ||
| SuperCLUE-V Models answer Chinese questions about visual inputs. | SuperCLUE | General visual questions | Everyday | Not in cards | GPT-4o 75.1 | Sep 2024 | |||
| SuperCLUE-Video Models generate videos from Chinese prompts. A judging model examines video frames and scores them against criteria for the prompt. | SuperCLUE team | Video generation | School | Not in cards | | Feb 2024 | Feb 2025 | ||
| SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail. | Shanghai Jiao Tong University | Fixing real repositories | Expert | Not in cards | GPT-5.4 Mini 21.5% | Jun 2026 | Apr 2026 | ||
| SWE-Game Agents build, complete, repair, or port games across 247 tasks derived from 41 reference games. Runtime probes and input replays check gameplay mechanics and preserved behavior. Vision-language rubrics separately grade presentation in construction tasks. | Shanghai Jiao Tong University, Zhongguancun Academy, Shanghai Innovation Institute, Shenzhen University, Beijing University of Posts and Telecommunications, and Elbetech Technology | Building whole projects | University | Not in cards | Claude Opus 5 64.0 | Sep 2026 | Aug 2026 | ||
| SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution. | Amazon | Fixing real repositories | Expert | Not in cards | Claude 3.5 Sonnet (October 2024) 14.1% | Apr 2025 | Jan 2025 | ||
| TakeoffBench-v1 Agents read residential blueprints and list material quantities by trade. A model judge matches items to expert estimates. The score combines coverage and quantity precision. | Handoff | Tasks from real jobs | Expert | Not in cards | Claude Fable 5 61.4% | Jul 2026 | Jun 2026 | ||
| Terminal-X A suite of three terminal datasets for coding agents. DeepTerminalBench tests single-shot depth, EvoCode-Bench tests multi-round iteration and RoadmapBench tests version upgrades on real repositories. | UniPat AI | Terminal tasks | Expert | Not in cards | Claude Opus 4.7 42.5% | May 2026 | Apr 2026 | ||
| TextVBench Models render requested text in video or preserve text from a reference image. The score tracks spelling, placement, and consistency across frames. | University of Science and Technology of China and collaborators | Video generation | Everyday | Not in cards | | Oct 2026 | May 2026 | ||
| ThermoQA Models calculate fluid properties and analyze thermodynamic components and cycles. A model extracts the answers, and a program checks values against reference tolerances. | Olivenet | Science questions | University | Not in cards | Claude Opus 4.6 94.1% | Mar 2026 | Mar 2026 | ||
| Tombo Full Transcript Models transcribe Portuguese parish registers from 1621 to 1737. The grader compares each full page with a reference transcription at the character level. | Mario Filho | Documents, charts and text in images | Expert | Not in cards | | Nov 2026 | Oct 2026 | ||
| TRACE Models identify reward hacking in agent trajectories from coding environments. | Patronus AI | Fixing real repositories | Expert | Not in cards | GPT-5.2 63% | Jan 2026 | Dec 2025 | ||
| TruthfulQA-Multi Models answer questions targeting common misconceptions in Basque, Catalan, Galician and Spanish. The translations support direct comparison with the English questions. | Blanca Calvo Figueras and colleagues | Short factual questions | Everyday | Not in cards | | Feb 2025 | Jul 2024 | ||
| Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. | Turing | Simulated companies | University | Not in cards | Claude Opus 4.7 60.2% | Jun 2026 | Apr 2026 | ||
| Turn based Benchmarks Models play turn based games including tic tac toe and rock paper scissors. The tasks evaluate game decisions and adaptation to opponents. | Mikael Aldy | Multi-agent games | Everyday | Not in cards | | Oct 2026 | Aug 2026 | ||
| UHGEval Models identify hallucinated Chinese text or continue a supplied passage. Classification uses accuracy, while generation uses BLEU-4, ROUGE-L, keyword precision, and BERTScore. | Renmin University of China, Institute for Advanced Algorithms Research and others | Hallucination rates | Everyday | Not in cards | GPT-4 0.70 | Nov 2023 | Jun 2024 | ||
| Vals Smith: schubydoo/clauster Agents modify schubydoo/clauster to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Luna 96.7% | Sep 2026 | Aug 2026 | ||
| VDC Benchmark Models describe the contents of a video in detail. The evaluation checks descriptions against human annotations using questions about the video. | Wenhao Chai and colleagues | Video captioning | Everyday | Not in cards | | Oct 2024 | Aug 2024 | ||
| Video Reality Test Vision-language models and humans judge whether ASMR videos with tight audio and visual coupling are real or AI-generated. It tests perceptual realism detection. | Video Reality Test authors | Video understanding | Everyday | Not in cards | | Dec 2025 | Nov 2025 | ||
| Video-Oasis A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions. | NAVER and collaborators | Video understanding | Everyday | Not in cards | | Mar 2026 | Oct 2025 | ||
| VWE-Bench Multimodal agents build interactive 3D worlds from a user request. They infer intent, plan the scene layout, call 3D tools for asset retrieval, editing and rendering, and revise from rendered feedback. The benchmark has 2,616 3D assets, 323 human-annotated seed worlds and 6,828 queries. Verified queries have ground truth, and the rest are scored with rubrics. | HKUST (Guangzhou) and Tencent | CAD models | Expert | Not in cards | GPT-5.5 57.3% | Aug 2026 | Aug 2026 | ||
| WBench Interactive video world models run 289 multi-turn cases with navigation, subject actions, event edits and perspective switches. 22 metrics cover quality, adherence, consistency and physics. | Meituan LongCat | Interactive world models | Everyday | Not in cards | | May 2026 | May 2026 | ||
| WildScore Models read score excerpts from Reddit music theory discussions and choose among answers drawn from the comments. | Gagan Mundada and collaborators | Documents, charts and text in images | University | Not in cards | Claude Sonnet 4.6 75% | Aug 2025 | Mar 2026 | ||
| WordEditBench Agents edit Word documents. A rule-based checker reads the document XML, rewards requested changes, and penalizes changes to unrelated content. | Tencent | Documents and spreadsheets | University | Not in cards | | Nov 2026 | Nov 2026 | ||
| τ-Rec Agents interview simulated moviegoers and search a catalog. A recommendation succeeds when it meets the user’s constraints and follows the service policy. | Bharath Sivaram Narasimhan and colleagues | General assistants | Everyday | Not in cards | | Jun 2026 | Apr 2026 | ||
| AACR-Bench Models review 200 real pull requests in many languages with full repository context. Their review comments are matched against expert-annotated issues. | Alibaba | Fixing real repositories | Expert | Not in cards | Claude Sonnet 4.5 16.1% | Jan 2026 | Dec 2025 | ||
| ACLUE Models read classical Chinese and answer questions about grammar, meaning, inference and cultural knowledge. All 15 tasks use multiple choice. | Yixuan Zhang and colleagues | Reading in many languages | University | Not in cards | chatglm2-6b 37.3% | Oct 2023 | Jun 2023 | ||
| AEC-Geometric-Bench Models locate doors, windows, and fixtures on construction drawings and trace walls and enclosed spaces. The scorer matches objects and compares wall and area masks. | Kamai | Documents, charts and text in images | University | Not in cards | | Aug 2026 | Aug 2026 | ||
| AfriHate Models classify hate speech and abusive language across African languages. | Shamsuddeen Hassan Muhammad and colleagues | Reading in many languages | Everyday | Not in cards | | Jan 2025 | Jul 2024 | ||
| AfriMed-QA Models answer about 15,000 open and multiple-choice medical questions from over 60 medical schools in 16 African countries. Questions span 32 specialties. | AfriMed-QA team (Olatunji et al., Intron Health and partners) | Medicine | University | Not in cards | | Nov 2024 | May 2024 | ||
| AgentScript-Eval Models write, debug, or optimize AgentScript definitions for Salesforce agents. A task passes when the program compiles and passes every execution test. | Salesforce | Function-level problems | University | Not in cards | Claude Opus 4.8 37% | Nov 2026 | Jun 2026 | ||
| AgentSWE Coding agents build, edit, or optimize agents across 25 software-engineering commissions. Frozen deliveries run on held-out cases, where execution gates and task-specific model-judged rubrics score behavior. Optimization tasks measure improvement against each source benchmark objective. | Beijing Academy of Artificial Intelligence, Institute of Automation of the Chinese Academy of Sciences, and Renmin University of China | Machine learning engineering | Expert | Not in cards | | Oct 2026 | Jul 2026 | ||
| AI Democracy Projects voter queries Models answer questions that US voters might ask about an election. Expert panels vote on whether each response contains inaccurate information. | Proof News and the Institute for Advanced Study | Long-form factuality | Everyday | Not in cards | GPT-4 19.2% | Feb 2024 | Dec 2023 | ||
| AI Engineering Lab simulation self-validation Agents predict how a mechanical system should behave, inspect simulation outputs, and decide whether the model is correct. The authors score these decisions with F1. | University of Innsbruck | Apps and MCP servers | University | Not in cards | GPT-4o 0.72 | Apr 2025 | Mar 2025 | ||
| AILA Statute Retrieval Models retrieve the Indian statutes that apply to a described legal situation. | Paheli Bhattacharya and colleagues | Law | University | Not in cards | | Oct 2020 | Oct 2026 | ||
| ALFWorld Agents complete household goals such as putting a washed apple in the fridge in a text world aligned with the ALFRED embodied simulator. Success is whether the goal state is reached. | ALFWorld team (Shridhar, Yuan, Côté, Bisk, Trischler, Hausknecht) | Simulated robots | Everyday | Not in cards | Claude 2 24.6% | Oct 2020 | Dec 2023 | ||
| AlphaEval Agent products such as Claude Code and Codex do 94 production tasks from 7 companies across 6 O*NET domains. Grading mixes formal checks, UI tests and rubrics. | AlphaEval authors (Lu et al.) | Running a business | Expert | Not in cards | Claude Opus 4.6 64.4% | Apr 2026 | Feb 2026 | ||
| Animation Bench Agents recreate web animations in HTML from reference frames and a network capture. The scorer compares appearance, motion and layout. | Physera | Web and app front ends | University | Not in cards | GPT-6.1 Sol 0.61 | Sep 2026 | |||
| Animation2Code Models recreate a web animation from a video using HTML, CSS, and JavaScript. The evaluator renders their code and compares appearance and motion separately. | University of California, Berkeley | Web and app front ends | University | Not in cards | GPT-5.4 0.84 | May 2026 | Mar 2026 | ||
| ArchDiagram Models create a draw.io diagram of a repository's architecture. A judge scores the content and readability across repeated attempts. | Drishti Peshwani | Fixing real repositories | University | Not in cards | Claude Sonnet 5.5 0.43 | Oct 2026 | Sep 2026 | ||
| ArtEdit-Bench-Lr Models retouch photos from English or Chinese instructions. GPT-4o scores instruction following and visual quality. Pixel errors measure fidelity to reference edits. | Tencent Hunyuan and Xiamen University | Apps and MCP servers | University | Not in cards | | Nov 2025 | |||
| Ask Ubuntu duplicate question retrieval Models retrieve previously posted Ubuntu support questions that ask about the same problem. | Tao Lei and colleagues | Questions that need search | University | Not in cards | | Dec 2015 | Oct 2026 | ||
| AssertLLM2 Models write assertions for hardware specifications. A formal verifier checks which assertions hold, which logic they cover, and which deliberately introduced faults they detect. | Hong Kong University of Science and Technology | Electronics | Expert | Not in cards | GPT-5.2 95% | May 2026 | Dec 2025 | ||
| AssetOpsBench Agents automate industrial asset work, such as condition monitoring, maintenance planning and intervention scheduling, using sensor data, failure models and work orders. | IBM Research | Other professions | Expert | Not in cards | GPT-4.1 65% | Jun 2025 | Apr 2025 | ||
| ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. | OpenCompass (Shanghai AI Laboratory) and collaborators | Science questions | Research frontier | Not in cards | GPT-5 42.9% | Nov 2025 | Aug 2025 | ||
| Augment Prism multi-turn coding benchmark Coding agents follow a queued conversation and edit a repository. A model compares the final changes with the original pull request. | Augment Code | Fixing real repositories | University | Not in cards | GPT-5.5 0.21 | May 2026 | Apr 2026 | ||
| AutoVend Agents buy stock and schedule deliveries for 20 simulated vending machines over 180 days. They share one warehouse and pay for each site visit. | StepFun and collaborators | Running a business | University | Not in cards | GPT-6.1 Sol 309.4 | Oct 2026 | Sep 2026 | ||
| AVGen-Bench Models generate videos with audio from prompts across eleven real-world categories. Specialist models and multimodal judges measure perceptual quality, synchronization, and adherence to detailed visual, speech, music, and physical requirements. | Microsoft Research | Video generation | University | Not in cards | | Apr 2026 | Mar 2026 | ||
| Aviation Blind Spot Benchmark Models answer questions about aviation incidents using a telemetry table. The harness checks whether they select the right records and derive the expected answers. | GasMan | Other professions | University | Not in cards | Claude Opus 4.7 0% | Nov 2026 | Aug 2026 | ||
| BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. | Mila | Simulated robots | Everyday | Not in cards | Claude 2 37.5% | Oct 2018 | Dec 2023 | ||
| BANKING77 Models assign an online banking question to one of 77 customer intents. | Iñigo Casanueva and colleagues | Finance | Everyday | Not in cards | | Mar 2020 | Oct 2026 | ||
| BEIR FiQA-2018 Retrieval Models rank 57,638 answer posts for 648 investment questions. BEIR sampled its test questions from the original FiQA training split. | BEIR team | Finance | Everyday | Not in cards | | Apr 2021 | Oct 2026 | ||
| BioPhys-Bridge Models connect observations in biophysics papers to quantitative explanations and cite supporting evidence. The authors score the cited evidence against reference identifiers. | Qingyang Xu | Genomics and life sciences | Expert | Not in cards | | May 2026 | Jun 2026 | ||
| BIRD-Interact Models solve ambiguous database tasks by talking with a simulated user, in a conversational mode and an agentic mode. 600 BI and CRUD tasks are checked by executable tests. | BIRD team (HKU) with Google Cloud | Data science and SQL | Expert | Not in cards | GPT-5 25.5% | Jun 2025 | Aug 2025 | ||
| Bridge LLM Bench Models make a contract bridge bid from a hand and the auction so far. The author compares each bid with two Standard American Yellow Card references. | albertogerli | Puzzles and chess | University | Not in cards | | Mar 2026 | Feb 2026 | ||
| Build-A-Chip-Bench Agents take a chip from workload specification to a routed GDS layout with open-source tools. Each design must pass hidden verification, routability and timing gates. | ChipAgents (with NVIDIA contributions) | Electronics | Research frontier | Not in cards | Claude Fable 5.1 16.6% | Oct 2026 | Sep 2026 | ||
| BuilderBench Agents learn to control a simulated robot that builds target block structures through physical interaction and exploration. | Princeton University | Embodied reasoning | Expert | Not in cards | GPT-6 Astra 98% | Oct 2025 | Sep 2026 | ||
| Business Evidence Judgment v4 Models label business claims as supported, contradicted, or unknown under three document conditions. The grader scores accuracy and compliance separately. | WangFuguang | Faithful to sources | University | Not in cards | Claude Haiku 4.5 90.7% | Oct 2026 | Apr 2026 | ||
| CADTestBench Models generate mechanical parts from text. Executable tests check the geometry and topology of each solid. A part passes when it satisfies every test. | Dimitrios Mallis and colleagues | CAD models | University | Not in cards | Claude Sonnet 4.6 81% | May 2026 | Feb 2026 | ||
| CAREBench Models answer 500 prompts with embedded child safety risks across 12 categories, such as grooming, impersonation and emotional dependency. A judge marks each response acceptable or not. | Handshake AI Research | Wellbeing and sycophancy | Everyday | Not in cards | Claude Fable 5 2.30% | Jun 2026 | Jun 2026 | ||
| ChartX Vision-language models do 7 chart tasks, such as question answering, summarization and redrawing, over 18 chart types and 22 topics. | Shanghai AI Laboratory (Xia, Zhang et al.) | Documents, charts and text in images | Everyday | Not in cards | GPT-4V 20.9 | Feb 2024 | Dec 2023 | ||
| CladBench v1 Models answer questions about building regulations, energy use, and retrofit decisions. The benchmark combines automatic checks with rubric scores from two model judges. | CladBrain | Tasks from real jobs | University | Not in cards | Claude Opus 4.7 0.89 | Aug 2026 | May 2026 | ||
| CLEF 2025 SimpleText Task 1 Models simplify individual sentences or whole Cochrane abstracts. The organizers tested new reviews and scored both tasks against plain English references. | University of Amsterdam and University of Western Brittany | Medicine | University | Not in cards | | Nov 2024 | Apr 2025 | ||
| CLI-Bench One-Shot Models submit complete files in one response for six CLI-Bench tasks. The original verifiers test debugging, implementation, and performance changes. | Arjun Shah | Function-level problems | University | Not in cards | gpt-oss-120b 100% | Oct 2026 | Sep 2026 | ||
| CLICK-ID Models distinguish clickbait headlines from other headlines published by Indonesian news outlets. | Andika William and colleagues | Reading in many languages | Everyday | Not in cards | | Oct 2026 | |||
| Clinical SOAP Note Evaluation: Safety-First Benchmarking Models turn 300 synthetic doctor-patient dialogues into clinical SOAP notes. Three judges check claims against transcript evidence and score numeric fidelity, coverage, and writing quality. The composite weights safety at 50%, coverage at 30%, and general quality at 20%. | Omi Health | Medicine | Expert | Not in cards | GPT-5.2 4.72 | Apr 2026 | Dec 2025 | ||
| ClutterBench Agents turn an article into an HTML graphic. A model grades the share of the drawn area occupied by unnecessary or oversized elements. | Adam Holter | Web and app front ends | University | Not in cards | Claude Sonnet 5.5 28.8% | Oct 2026 | Oct 2026 | ||
| CMMU Models answer Chinese multimodal questions across seven school subjects. | BAAI FlagEval | General visual questions | School | Not in cards | GPT-4V 30.9 | Jan 2024 | Jan 2024 | ||
| CodeCompass Models solve competitive programming problems. Test cases check the generated programs. | OpenCompass and collaborators | Competitive programming | Competition | Not in cards | | Jul 2025 | Apr 2025 | ||
| CodeIPI Coding agents fix bugs while handling untrusted repository text. The benchmark scores task completion, resistance to embedded instructions and false alarms on clean tasks. | Inspect Evals contributors | Jailbreaks and prompt injection | University | Not in cards | Claude Sonnet 4.6 91.1% | Apr 2026 | Feb 2026 | ||
| Collab-Overcooked Two agents exchange messages and use different kitchen equipment to fill orders. The benchmark measures task completion and progress through required cooking steps. | Haochen Sun and colleagues | Multi-agent games | Everyday | Not in cards | Claude Sonnet 4 86.7% | Feb 2025 | May 2025 | ||
| Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. | UC Berkeley, Snorkel AI and collaborators (Asawa, Orlanski et al.) | Learning from the context | Expert | Not in cards | Claude Sonnet 4.6 0.20 | Jun 2026 | Apr 2026 | ||
| Corr2Cause Models decide whether a causal claim follows from statements about correlations and independences among variables. The original study reports F1 for detecting valid claims. | CausalNLP and collaborators | Language reasoning | University | Not in cards | | May 2023 | Jul 2023 | ||
| CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. | ETH Zurich (SPY Lab) + Anthropic + University of Haifa + Tel Aviv University | Finding and exploiting real bugs | Research frontier | Not in cards | Claude Mythos 5 34.2% | Jul 2026 | Jun 2026 | ||
| CutCraft Benchmark Models generate multi-shot videos under editing instructions. The evaluator checks transitions, shot structure, and the timing of audio relative to cuts. | Alibaba Group and collaborators | Video generation | University | Not in cards | | Sep 2026 | Mar 2026 | ||
| CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. | UC Berkeley RDI (Sunblaze) and collaborators | Finding and exploiting real bugs | Expert | Not in cards | Claude Opus 5 89% | Jun 2026 | Jul 2026 | ||
| Dan’s Editorial Checks Models edit articles and provide literary feedback. Every scores their work against the author’s editorial checks. | Every | Taste and style | University | Not in cards | GPT-6.1 Sol 86% | Sep 2026 | |||
| Data Agent Benchmark Agents answer business questions over data split across several database systems, with duplicated and inconsistent records and key facts buried in text. The 104 queries cover 17 datasets and four database systems, based on a study of companies building data agents in six industries. Scores are pass@1 against known answers. | UC Berkeley (EPIC Data Lab) and Hasura PromptQL | Data science and SQL | Expert | Not in cards | | Mar 2026 | Aug 2026 | ||
| DataPrep-Bench Finance Models prepare training data for the finance domain. The score is the published downstream average after backbone training. | Peking University | Machine learning engineering | Expert | Not in cards | GPT-5.3 Codex 59.3 | May 2026 | Feb 2026 | ||
| DataPrep-Bench General Models prepare training data for the general domain. The score is the published downstream average after backbone training. | Peking University | Machine learning engineering | Expert | Not in cards | Claude Opus 4.6 78.2 | May 2026 | Feb 2026 | ||
| DataPrep-Bench Law Models prepare training data for the law domain. The score is the published downstream average after backbone training. | Peking University | Machine learning engineering | Expert | Not in cards | GPT-5.2 76.4 | May 2026 | Feb 2026 | ||
| DataPrep-Bench Math Models prepare training data for the math domain. The score is the published downstream average after backbone training. | Peking University | Machine learning engineering | Expert | Not in cards | GPT-5.2 25.5 | May 2026 | Feb 2026 | ||
| DataPrep-Bench Medical Models prepare training data for the medical domain. The score is the published downstream average after backbone training. | Peking University | Machine learning engineering | Expert | Not in cards | Claude Opus 4.6 40.5 | May 2026 | Feb 2026 | ||
| DataPrep-Bench Science Models prepare training data for the science domain. The score is the published downstream average after backbone training. | Peking University | Machine learning engineering | Expert | Not in cards | | May 2026 | Feb 2026 | ||
| DBpedia 14 Models classify English descriptions of entities into fourteen categories from the DBpedia ontology. | Xiang Zhang and colleagues | Reading comprehension | School | Not in cards | | Sep 2015 | Oct 2026 | ||
| DecodingTrust Models are tested on eight trustworthiness areas, including toxicity, stereotype bias, robustness, privacy, machine ethics and fairness. | UIUC Secure Learning Lab (Bo Li) and coauthors | Bias, fairness and politics | Everyday | Not in cards | | Jun 2023 | Dec 2023 | ||
| Delulu Models fill missing code in repository contexts across seven programming languages. Per-sample containers check compilation or type checking, while separate metrics compare completions with valid and hallucinated references. | Microsoft | Function-level problems | University | Not in cards | | May 2026 | Nov 2024 | ||
| DiaBLa Models translate informal conversations between English and French while preserving meaning across turns. | Rachel Bawden and colleagues | Translation | School | Not in cards | | May 2019 | Oct 2026 | ||
| DRBench Agents investigate open-ended business questions across enterprise applications and produce reports. The evaluator checks whether the reports recover relevant information and support their claims. | Amirhossein Abaskohi and colleagues | Deep research | University | Not in cards | GPT-5 63.8 | Sep 2025 | Aug 2025 | ||
| EdgeBench Agents iterate for up to 12 hours on 134 executable real-world tasks with multi-level feedback. Scores are tracked at 2 hour steps to measure learning from the environment. | ByteDance Seed | Long terminal sessions | Expert | Not in cards | Claude Opus 4.8 51.3 | Jul 2026 | May 2026 | ||
| Ego2Act Models generate first-person video showing how to complete a task from an initial image. A judge checks the steps and their physical plausibility. | Mohamed bin Zayed University of Artificial Intelligence and collaborators | Video generation | Everyday | Not in cards | | Oct 2026 | May 2026 | ||
| EgyHellaSwag Models select a plausible ending for a situation described in Egyptian Arabic. The evaluation adapts HellaSwag to the dialect. | Abdellah El Mekki and colleagues | Reasoning in many languages | Everyday | Not in cards | | May 2025 | Apr 2025 | ||
| EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. | Scale AI + Center for AI Safety + MIT | General visual questions | Competition | Not in cards | Claude Fable 5 39.3% | Feb 2025 | Jul 2026 | ||
| Estonian Valence Corpus Models predict the emotional valence of Estonian news text using human annotations. | Hille Pajupuu and colleagues | Reading in many languages | Everyday | Not in cards | | Oct 2026 | |||
| FactoryBench (Forgis Labs) Models interpret sensor readings from industrial robot arms and predict how the machines respond to changes. They also recommend actions when the readings indicate a fault. | Forgis Labs | Other professions | Expert | Not in cards | Claude Sonnet 4.6 36% | May 2026 | Feb 2026 | ||
| Faroese Metaphorical Explanations Models read a Faroese word in figurative use and select its meaning from four dictionary explanations. | University of the Faroe Islands and University of Copenhagen | Reading in many languages | Everyday | Not in cards | GPT-5 88.4% | May 2026 | Aug 2025 | ||
| FeatureBench Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically. | LiberCoders | Fixing real repositories | Expert | Not in cards | GPT-5.1-Codex 60.2% | Feb 2026 | Dec 2025 | ||
| FeatureBench Lite The public 30-task Lite leaderboard ranks passed-test percentages and reports fully resolved-task percentages separately. | LiberCoders | Fixing real repositories | Expert | Not in cards | GPT-5.1-Codex 60.2% | Feb 2026 | Dec 2025 | ||
| Financial PhraseBank Models classify the financial sentiment of a news sentence as positive, negative, or neutral. | Pekka Malo and colleagues | Finance | University | Not in cards | | Jul 2013 | Oct 2026 | ||
| FinDendro Models recall dated US financial facts and identify the limits of their knowledge. Four prompt styles test value recall, self-knowledge, and historical timing. | Lingfei Qian and collaborators | Finance | Expert | Not in cards | GPT-5 0.76 | Nov 2026 | Feb 2026 | ||
| FinRED Financial Safety Models answer prompts about financial risks. A finance-specific judge grades each response and reports the percentage that assist harmful conduct. | DATUMO, Financial Security Institute, and KAIST | Deception and alignment | University | Not in cards | GPT-5 0.15% | Jun 2026 | Aug 2025 | ||
| FLAMES Models respond to Chinese prompts about safety and values. Evaluators assess the responses against the benchmark's alignment criteria. | Shanghai AI Laboratory and collaborators | Deception and alignment | Everyday | Not in cards | Baichuan2-13B-Chat 43.2% | Nov 2023 | Oct 2023 | ||
| Founder Bench Agents run four matched businesses per model through the acoco platform. The evaluation tracks real advertising, customer attention, and website traffic after seven days. Its headline customer-impressions metric adds ad impressions to organic site visits. | Eico | Running a business | Expert | Not in cards | Claude Fable 5 127902.0 | Jul 2026 | Jul 2026 | ||
| Frontier Agentic STEM Agents produce scientific results in a terminal. Task verifiers recompute the expected results from the underlying data. | Turing | Research science | Expert | Not in cards | GPT-5.6 Sol 55.1% | Aug 2026 | Jul 2026 | ||
| FrontierCAD Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry. | Normal Research | CAD models | Expert | Not in cards | GPT-6 Astra 0.67 | Oct 2026 | Sep 2026 | ||
| FrontierEE Bench Agents run electrical engineering experiments on real lab hardware through networked instruments, camera photos and tickets to a human operator. Deterministic verifiers check 10 tasks against protected bench records. | Evident Machines | Electronics | Expert | Not in cards | Claude Fable 5.2 36.7% | Nov 2026 | Nov 2026 | ||
| GateMem Memory agents serve several people who share one memory pool in hospitals, offices, schools and homes. Each long episode checks three things: answering legitimate requests after state updates, refusing information a person is not allowed to see, and forgetting records after an explicit deletion request. Hidden checkpoints and leak targets let a structured judge score each response. | GateMem authors (Zhe Ren and coauthors) | Learning from the context | Everyday | Not in cards | | Jun 2026 | Apr 2026 | ||
| GEN-Bench Agents select job operations as conditions change in a simulated factory. The benchmark compares schedule completion times with the best known result for each instance. | Shijie Cao and Yuan Yuan | Tasks from real jobs | University | Not in cards | | Aug 2025 | Dec 2025 | ||
| GeoBiaset Models predict photo coordinates before and after researchers insert a person into the scene. The coordinates stay fixed across all edits of each background. | University of Florence | Bias, fairness and politics | University | Not in cards | GLM-4.1V-9B-Thinking 69.2% | Oct 2025 | |||
| Greek Legal Code Models classify Greek legal texts by subject using their place in the Greek Legal Code hierarchy. | Christos Papaloukas and colleagues | Law | University | Not in cards | | Sep 2021 | Oct 2026 | ||
| Gulf Text-to-SQL Bench Models write SQL for the same 30 business questions in English, Modern Standard Arabic, and Gulf Arabic. The harness executes each query. | Usman Ahmad | Data science and SQL | University | Not in cards | | Nov 2026 | Sep 2026 | ||
| Gym-Anything-RT Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time. | Carnegie Mellon University (Welleck lab, CMU L3) | Video games | Everyday | Not in cards | GPT-6 Astra 9.00% | Oct 2026 | Sep 2026 | ||
| H2R-Bench Models turn first-person demonstrations into videos of a specified robot performing the task. The benchmark checks contacts, object responses, and completion of the goal. | Dingyi Rong and colleagues | Video generation | Everyday | Not in cards | | Aug 2026 | Mar 2026 | ||
| HackSynth OverTheWire Agents complete scored security exercises from OverTheWire. The published benchmark covers 80 challenges, and the hub port selects 46. | HackSynth authors | Capture-the-flag challenges | Expert | Not in cards | | Dec 2024 | Aug 2024 | ||
| HackSynth PicoCTF Models attempt 120 tasks selected from picoCTF. The benchmark measures the percentage of tasks completed, and the hub port checks submitted answers against reference flags. | Eötvös Loránd University | Capture-the-flag challenges | Competition | Not in cards | | Dec 2024 | Aug 2024 | ||
| HarnessOpt-Bench Coding agents edit a target agent’s harness under a fixed evaluation budget. The grader measures improvement over the starting harness on held-out tasks. | Scale AI | Machine learning engineering | Expert | Not in cards | Claude Opus 5 0.47 | Aug 2026 | Jul 2026 | ||
| HealthBench Meta Models judge whether a medical response meets a specific rubric item. The evaluation compares each binary judgment with physician labels and reports agreement metrics. | OpenAI | Medicine | Expert | Not in cards | GPT-4.1 0.71 | May 2025 | Apr 2025 | ||
| Hold or Fold: AI Under Pushback Models answer questions and face three kinds of pushback. The grader counts how often a model keeps a correct answer across 72 cases. | Sundas N | Wellbeing and sycophancy | School | Not in cards | | Oct 2026 | Sep 2026 | ||
| HybridDeepResearch Agents combine database queries and web search to answer research questions under constraints. | Snowflake and collaborators | Data science and SQL | Expert | Not in cards | | Aug 2026 | Feb 2026 | ||
| IberoBench Models perform language understanding and generation tasks in Iberian languages. The suite combines existing evaluations with new tests specific to each language. | Irene Baucells and colleagues | Reasoning in many languages | University | Not in cards | | Jan 2025 | Apr 2024 | ||
| ICLayout-Bench Agents turn transistor netlists into chip layouts. Design tools check the layout rules and connectivity, then measure electrical performance and area. | Zhangmai Li and colleagues | Electronics | Expert | Not in cards | GPT-6 Astra 80.3 | Oct 2026 | |||
| IlùBench Models explain each Igbo proverb twice, once after an English prompt and once after an Igbo prompt. The board records language and cultural register separately. | Chuma B. Chukwu Jr., UUAMNI | Chat and instructions in many languages | Everyday | Not in cards | Claude Fable 5.1 100% | Jul 2026 | Sep 2026 | ||
| IN22-Gen Models translate parallel sentences across English and Indian languages. Google evaluates whether an embedding model can match a sentence with its translation. | Jay Gala and colleagues | Translation | School | Not in cards | | May 2023 | Oct 2026 | ||
| INJONGO Models identify a speaker’s intent and extract information from utterances in 16 African languages. | Hao Yu and colleagues | Reading in many languages | Everyday | Not in cards | GPT-4o 65.1% | Feb 2025 | Dec 2024 | ||
| Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. | Apollo Research | Deception and alignment | University | Not in cards | davinci-002 4.67% | Nov 2023 | Sep 2023 | ||
| JSONSchemaBench Models or constrained decoders generate outputs under real JSON schemas. The evaluation tests schema coverage and valid generation. | Saibo Geng and colleagues | Following precise instructions | University | Not in cards | GPT-4o 91.4% | Jan 2025 | Mar 2025 | ||
| JudgerBench V2 Judge models evaluate responses across tasks. OpenCompass reports results for generalist judges on the revised benchmark. | OpenCompass and collaborators | Other | University | Not in cards | | Jul 2025 | Apr 2025 | ||
| JudgerBench Original Replaced in Jul 2025 | Not in cards | GPT-4o 0.82 | Oct 2024 | Sep 2024 | |||||
| KellyBench Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth. | General Reasoning | Social simulation | Expert | Not in cards | GPT-5.4 -7.90% | Apr 2026 | Mar 2026 | ||
| KUMO Agents make queries and use the responses to identify hidden facts in generated reasoning games. Harbor offers 5,050 easy games and 250 hard games. | KUMO team | Puzzles and chess | University | Not in cards | | Apr 2025 | Mar 2025 | ||
| LABBench2 CloningQA Models answer molecular biology questions using supplied information or retrieval tools. | Edison Scientific | Research science | Expert | Not in cards | | Feb 2026 | Feb 2026 | ||
| LABBench2 FigQA2 (image) Models answer research questions about figures supplied as images. | Edison Scientific | Research science | Expert | Not in cards | GPT-5.2 66.3% | Feb 2026 | Feb 2026 | ||
| LABBench2 FigQA2 (PDF) Models answer research questions about figures supplied as PDF files. | Edison Scientific | Research science | Expert | Not in cards | GPT-5.2 64.4% | Feb 2026 | Feb 2026 | ||
| LABBench2 PatentQA Models locate and interpret evidence in patent documents. | Edison Scientific (formerly FutureHouse) | Research science | Expert | Not in cards | GPT-5.2 Pro 90.9% | Feb 2026 | Feb 2026 | ||
| LABBench2 SourceQuality Models assess the quality of scientific information sources. | Edison Scientific (formerly FutureHouse) | Research science | Expert | Not in cards | | Feb 2026 | |||
| LABBench2 TableQA2 (image) Models answer research questions about tables supplied as images. | Edison Scientific | Research science | Expert | Not in cards | | Feb 2026 | Feb 2026 | ||
| LABBench2 TableQA2 (PDF) Models answer research questions about tables supplied as PDF files. | Edison Scientific | Research science | Expert | Not in cards | | Feb 2026 | Feb 2026 | ||
| LABBench2 TrialQA Models use clinical trial records to answer research questions. | Edison Scientific (formerly FutureHouse) | Research science | Expert | Not in cards | GPT-5.2 Pro 93.3% | Feb 2026 | Feb 2026 | ||
| LeanLean Agents shorten Lean projects while preserving protected theorem statements and passing the build and axiom checks. | MathArena | Formal proofs | Expert | Not in cards | GPT-6.1 Sol 50.5% | Nov 2026 | Oct 2026 | ||
| LLM-CAE-Benchmark Models generate Gmsh geometry and Elmer solver files for a square bar and a wheel and axle assembly. The authors score each output separately. | University of Birmingham | Scientific code | University | Not in cards | | Jun 2025 | Apr 2024 | ||
| Long Code Arena Models use project-wide context to generate and complete code, repair builds, find bugs, write commit messages, and summarize modules. Task scores use build success, exact match, API recall, text similarity, localization F1, or a model-based documentation comparison. | JetBrains Research and academic collaborators | Fixing real repositories | University | Not in cards | o1 0.96 | Jun 2024 | Dec 2024 | ||
| Long-Horizon Freight AuditBench Challenge Agents price 100 difficult freight invoices and flag incorrect charges. The cases include incomplete information and exceptions to ordinary contract terms. | Loop | Tasks from real jobs | Expert | Not in cards | Claude Opus 5 10% | Sep 2026 | Aug 2026 | ||
| Long-Horizon Freight AuditBench Production Agents price 200 freight invoices from contracts and carrier documents, then flag billing errors. Loop weights accuracy to reflect a 24 percent billing error rate. | Loop | Tasks from real jobs | University | Not in cards | Claude Opus 5 16.1% | Sep 2026 | Aug 2026 | ||
| Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. | Jasmine Brazilek and colleagues | Deception and alignment | Everyday | Not in cards | Claude Sonnet 4.6 3.23 | Jul 2026 | May 2026 | ||
| MANTA (Animal Welfare) Models discuss animal welfare across five conversational turns. A judge measures whether they recognize the concern and maintain their position under pressure. | Isabella Luong and colleagues | Deception and alignment | Everyday | Not in cards | Claude Opus 4.7 0.76 | Apr 2026 | Apr 2026 | ||
| Medical Speech-to-Text Benchmark Speech models transcribe 1,513 English medical recordings. A common scorer measures word errors, medical-term errors, drug-name errors, and dosage matches. Separate tests measure batch response time and final streaming latency. | Omi Health | Speech recognition and translation | Expert | Not in cards | GPT-Live-Transcribe 1.32% | Sep 2026 | Sep 2026 | ||
| MentalChat16K Counseling Evaluation Models answer 200 mental health questions from online forums. Judges rate each response from one to ten on seven counseling skills. | University of Pennsylvania and colleagues | Medicine | Expert | Not in cards | | Jan 2024 | Dec 2023 | ||
| metabench Models answer selected items from established evaluations. The benchmark uses a sparse item set to estimate reasoning and knowledge with fewer questions. | Alex Kipnis and colleagues | Composite indices | University | Not in cards | | Jul 2024 | May 2024 | ||
| Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. | Google DeepMind | Video understanding | Expert | Not in cards | GPT-5 37.6% | May 2026 | Sep 2025 | ||
| Mini Amusement Parks Agents build rides, hire staff, and manage an amusement park. The simulator scores the park’s final value. | Skyfall AI | Video games | Everyday | Not in cards | GPT-5 371375.0 | Nov 2025 | Sep 2025 | ||
| Mitosis Bench Agents mark every dividing cell in a tumor image. Each mark must fall within 30 pixels of an expert label. The grader rejects extra or missing marks. | Dylan Tirandaz | Medicine | Expert | Not in cards | | Oct 2026 | Sep 2026 | ||
| MK-Bench Agents schedule jobs when machines break down or orders arrive or disappear. The grader compares each completion time with the best known schedule. | Shijie Cao and Yuan Yuan | Tasks from real jobs | University | Not in cards | | Jan 2026 | Dec 2025 | ||
| MMDU Vision-language models hold long conversations about many images, up to 20 images and 27 turns per dialog. A GPT-4o judge scores the answers. | Shanghai AI Laboratory and collaborators (Liu, Zang et al.) | General visual questions | Everyday | Not in cards | Claude 3 Opus 62.6 | Jun 2024 | Apr 2024 | ||
| MobileGym Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state. | MobileGym authors | Phones | Everyday | Not in cards | | May 2026 | Mar 2026 | ||
| MoI-Bench Models generate and edit images for mobile users. The authors compare image quality across 64 subtasks and measure the cost of running models on phones. | Kailai Feng and colleagues | Image generation and editing | Everyday | Not in cards | GPT-Image-2 0.97 | Oct 2026 | Feb 2026 | ||
| MPIE-Bench Image editing models place several named people into shared contact actions such as an embrace or a carry. Scoring targets fused limbs, extra extremities and bodies passing through each other. | muset.ai | Image generation and editing | Everyday | Not in cards | | Jul 2026 | May 2025 | ||
| MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. | Chiyeong Heo and colleagues | Video understanding | Expert | Not in cards | GPT-5.2 16.2% | May 2026 | Feb 2026 | ||
| MultipanelVQA Models answer questions about images arranged into panels. The benchmark tests whether a model can locate the relevant panels and compare their contents. | Yue Fan and colleagues | General visual questions | Everyday | Not in cards | | Jan 2024 | Jan 2024 | ||
| Navital Gym Agents complete tasks in six simulated Indian service settings. This port covers UPI Desk, which grades 50 payment and dispute tasks using written rules. | Navital Labs | Apps and MCP servers | Everyday | Not in cards | Claude Opus 5.5 100% | Oct 2026 | Sep 2026 | ||
| NCP-Bench Models narrate interactive stories while preserving facts, commitments, and plot order against player interventions. A model auditor checks 100 movie-derived environments after each turn. The evaluation reports survival, conflicts, and narrative progress. | University of Macau, Westlake University, Harbin Institute of Technology and others | Multi-turn conversation | Everyday | Not in cards | GPT-5.2 32.9 | Aug 2026 | Jan 2026 | ||
| NoisyBench Models answer tasks from 11 existing datasets in RAG, reasoning, alignment and tool use while their context holds distractors, such as random documents, unrelated chat history and hard negatives. The score shows how much accuracy drops from the clean version of each task. | KAIST AI | Faithful to sources | Everyday | Not in cards | | Jan 2026 | Sep 2025 | ||
| Omni-InteractScope Explicit Request Models answer explicit requests during audio and video streams. The score is the published average across tasks T1 through T7. | Alibaba Group (Qwen, Alibaba Token Hub) + Hong Kong Polytechnic University | Video understanding | Everyday | Not in cards | | Oct 2026 | May 2026 | ||
| OmniPhys Models solve Chinese physics problems from middle school through university and draw diagrams. The evaluator grades answers, reasoning, and generated figures. | East China Normal University and collaborators | Science questions | University | Not in cards | | Aug 2026 | Dec 2025 | ||
| OmniVBench Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly. | Tencent | Video generation | Everyday | Not in cards | | Sep 2026 | |||
| OptimBench Dispatch Agents route vehicles under capacity and time constraints, then revise their routes after a disruption. | yferc | Tasks from real jobs | University | Not in cards | Claude Opus 5 0.92 | Jul 2026 | Jul 2026 | ||
| ORCA-bench Agents investigate simulated production incidents and identify their root causes. The benchmark contains 1,079 tasks, including 324 held-out tasks covering 31 incidents. | ORCA-bench team | Terminal tasks | Expert | Not in cards | Claude Opus 4.7 29.6% | Jul 2026 | Apr 2026 | ||
| Parashift Document AI Benchmark Models extract fields from 3,365 business documents across five datasets. Parashift compares predictions with human-verified values using a string similarity threshold of 0.8. | Parashift | Documents, charts and text in images | Everyday | Not in cards | Claude Opus 5.5 85.3% | Sep 2026 | Sep 2026 | ||
| PDEAgent-Bench Models write finite element solvers for partial differential equations. A solver must run successfully, meet a numerical error tolerance, and finish within its time limit. | PDEAgent-Bench team | Scientific code | Expert | Not in cards | | May 2026 | Apr 2026 | ||
| Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. | Robocurve | Jailbreaks and prompt injection | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. | Shanghai AI Laboratory, Shanghai Jiao Tong University, and collaborators | Image generation and editing | Everyday | Not in cards | | Oct 2025 | |||
| PLCWorld Models write Structured Text programs for motion control and material handling. A plant simulator measures task completion and safety violations separately. | Dongguk University and MOAI technologies | Electronics | University | Not in cards | GPT-5.5 62.6% | Oct 2026 | Sep 2026 | ||
| Prosus Vending Bench v2 Agents run six simulated vending machines across three locations for 30 days. They order stock, set prices, and collect cash. The score measures final bank balance in euros. | Prosus AI | Running a business | University | Not in cards | Claude Opus 5 4497.2 | Sep 2026 | Jul 2026 | ||
| ProtocolLLM Models write SystemVerilog modules for four communication protocols. The authors check syntax, synthesis, and whether simulated signals meet protocol timing requirements. | University of Illinois Urbana-Champaign and CISPA Helmholtz Center for Information Security | Electronics | University | Not in cards | | Jun 2025 | Apr 2026 | ||
| PunEval Models recognize, explain and generate English puns. The evaluation checks recognition against labels and compares explanations with human annotations. | Fudan University | Language reasoning | Everyday | Not in cards | GPT-3.5 Turbo 1.00 | Apr 2024 | Feb 2024 | ||
| ReactHuman Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate. | ReactHuman authors | Simulated robots | Everyday | Not in cards | GPT-5.5 63.7% | Sep 2026 | May 2026 | ||
| Real World VoiceEQ Voice systems recognize, produce and respond to acoustic information across speech recognition, speech to speech, speech understanding and text to speech tasks. | Hume AI | Audio understanding and voice | Expert | Not in cards | | Jul 2026 | Sep 2026 | ||
| realitybench Models build web interfaces for twelve browser tasks. Tests exercise normal use, server errors, invalid input, repeated clicks, and small screens. | Faisal Ahmad | Web and app front ends | University | Not in cards | | Oct 2026 | Nov 2026 | ||
| RH SWE-bench 1.1 Agents repair Go and Python code in Red Hat ecosystem repositories. The current candidate contains 337 tasks across 24 repositories after review and deduplication. | RH SWE-bench team | Fixing real repositories | Expert | Not in cards | Oct 2026 | ||||
| RH SWE-bench v1 Replaced in Oct 2026 | Red Hat Emerging Technologies | Not in cards | Claude Opus 4.6 63.3% | Jul 2026 | Apr 2026 | ||||
| RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. | Satvik Sharma and colleagues | Simulated robots | Everyday | Not in cards | | Nov 2026 | Sep 2026 | ||
| RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. | RobotWorld team | Simulated robots | Everyday | Not in cards | GPT-6 Astra 19.1% | Oct 2026 | Sep 2026 | ||
| RQA-70K weather radar forecast evaluation Models compare forecast radar images or sequences with observations. They rate errors such as missed storms and false alarms and write assessments of forecast quality. | Shanghai Artificial Intelligence Laboratory and collaborators | Lab and data work | University | Not in cards | o1 52.7% | Aug 2025 | Feb 2025 | ||
| RTL-OPT Models rewrite Verilog to improve a circuit while preserving its behavior. The authors compare synthesized power, timing, and area against human reference designs. | Hong Kong University of Science and Technology | Electronics | University | Not in cards | DeepSeek-Coder-V2-Lite-Instruct 72.5% | Apr 2026 | |||
| RTLLM 2.0 Models write Verilog for 50 digital designs from natural language specifications. Testbenches check the implementation before synthesis measures power, timing, and area. | Shang Liu and colleagues | Electronics | University | Not in cards | | Oct 2024 | Apr 2026 | ||
| RTLLM 1.0 Replaced in Oct 2024 | Yao Lu and colleagues | Not in cards | Aug 2023 | ||||||
| S-EMBER Models answer memory questions from first-person video using only frames available before the query time. The benchmark scores multiple-choice accuracy and temporal overlap between predicted evidence intervals and annotated intervals. | Meta FAIR and Reality Labs | Long videos | Everyday | Not in cards | | Jul 2026 | Mar 2026 | ||
| ScholarCatalyst Given a research question, retrieve the 'catalyst papers' that 184 lead authors labeled for 207 projects, searching only prior literature. Recall@20. | Stanford + collaborators | Research science | Expert | Not in cards | Claude Fable 5.1 53% | Oct 2026 | Sep 2026 | ||
| SciMMIR Models match figures and tables from scientific papers with their descriptions. | Siwei Wu and colleagues | Documents, charts and text in images | Expert | Not in cards | | Jan 2024 | Nov 2023 | ||
| ScreenASR-Bench Models transcribe speech using on-screen text, visible objects, or clues from the scene. The authors report word error rate and recall. | Mingwei Fu and colleagues | Speech recognition and translation | Everyday | Not in cards | | Sep 2026 | Apr 2026 | ||
| Sherlock Holmes Q&A Models search 55 Sherlock Holmes stories and answer 140 literary comprehension questions. A model judge checks their answers. | Alleinzellgaenger | Reading comprehension | School | Not in cards | GPT-5 100.0 | Oct 2025 | Aug 2025 | ||
| SimulCost Agents tune physics simulations to meet an accuracy target. The authors measure success and computational cost for an initial guess and for repeated attempts with feedback. | Yadi Cao, Sicheng Lai, and colleagues | Research science | Expert | Not in cards | GPT-5 80.8% | Feb 2026 | Aug 2025 | ||
| Sparse Fourier Recovery Models reconstruct a sparse one-dimensional signal from incomplete Fourier measurements. A reference solver provides a comparison for reconstruction accuracy. | Verifiable Labs | Scientific code | Expert | Not in cards | Claude Haiku 4.5 0.36 | Apr 2026 | Mar 2026 | ||
| Spider2-V Multimodal agents do data science and engineering workflows in real enterprise apps such as BigQuery, dbt and Airbyte, on a desktop computer. Execution-based checks score each task. | HKU XLANG Lab and collaborators | Desktop apps | Expert | Not in cards | GPT-4 Turbo 14% | Jul 2024 | May 2024 | ||
| SpireBench (HermesBridge trial-v0) Models play Slay the Spire 2 from structured game states. Each plays all five characters at the base difficulty, with one run per character. | Kareem | Video games | Everyday | Not in cards | GPT-5.5 28.0 | Apr 2026 | Apr 2026 | ||
| StrataBench Land v1 Models classify land instruments, extract lease terms, and distinguish petroleum terms in three separate benchmarks. Classify and Extract have their own records. Define publishes three measures without one headline score. | Collide AI | Law | Expert | Not in cards | GPT-5.6 Luna 0.94% | Nov 2026 | Jul 2026 | ||
| StrataBench Land v1 Classify Models identify the legal act performed by each of 66 synthetic land instruments. Collide scores label accuracy under open and closed prompt protocols. | Collide AI | Law | Expert | Not in cards | GPT-5.6 Luna 93.9% | Nov 2026 | Jul 2026 | ||
| StrataBench Land v1 Extract Models extract ten shared lease fields from 628 synthetic instruments. Collide divides correct field decisions by correct, omitted, and wrong decisions, excluding unscored fields. | Collide AI | Law | Expert | Not in cards | | Nov 2026 | Jul 2026 | ||
| SuperCLUE-Agent Scenario Agents complete tasks drawn from Chinese application scenarios. | SuperCLUE | General assistants | University | Not in cards | | Oct 2026 | |||
| SuperCLUE-Code Scenario Models write code for application scenarios. | SuperCLUE | Building whole projects | University | Not in cards | GLM-5.3-Flash 86.7 | Oct 2026 | |||
| SWE-Bench ProMax Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions. | ByteDance | Fixing real repositories | Expert | Not in cards | GPT-5.2 41.2% | Aug 2026 | Feb 2026 | ||
| SWE-Interact Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session. | Scale AI | Fixing real repositories | Expert | Not in cards | Claude Opus 4.8 50.7% | Jun 2026 | May 2026 | ||
| SynthDocBench Vision-language models answer questions about synthetic long documents where length, layout, modality and difficulty are varied one at a time to locate failure causes. | ServiceNow | Documents, charts and text in images | Everyday | Not in cards | | Jul 2026 | Mar 2026 | ||
| Tatoeba sentence retrieval Models find the English translation of a sentence among competing sentences using multilingual embeddings. | Facebook AI Research | Translation | School | Not in cards | | Dec 2018 | Oct 2026 | ||
| TeachObs Models identify teaching practices in classroom video and assess entire lessons. Human annotations cover 39 observable behaviors in 5,158 segments from eight countries. | Indiana University Bloomington and collaborators | Other professions | University | Not in cards | Claude Opus 4.7 0.28 | May 2026 | Apr 2026 | ||
| TheoPhysEval Agents solve 50 unpublished research-level theoretical physics problems on their own, with literature search and scripts. Each final answer is checked against an expert answer, and the score is the pass rate. | ICTS Bengaluru (Suvrat Raju) with 35 researchers incl. Google DeepMind, Harvard and KU Leuven | Research science | Research frontier | Not in cards | GPT-6 Astra 98% | Oct 2026 | Sep 2026 | ||
| Toloka Arena Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed. | Toloka | Customer service | Expert | Not in cards | Claude Opus 4.6 33% | Apr 2026 | Mar 2026 | ||
| TRAIL Models read long agent workflow traces and must find and classify the errors in them, using a formal taxonomy of agent error types. The set has 148 large human-annotated traces from established agent benchmarks. | Patronus AI | General assistants | Expert | Not in cards | | May 2025 | May 2025 | ||
| TrialDesignBench Models select a dose-finding design for an early oncology trial. The authors compare their choices with expert reference designs across 617 trials. | Yihan Hank Tang and colleagues | Medicine | Expert | Not in cards | GPT-5 0.52 | Nov 2026 | Jul 2026 | ||
| TriPhysBench Models answer quantitative questions about physical events in videos. Some questions require measuring an object or looking up its specifications. | Renmin University of China and collaborators | Embodied reasoning | School | Not in cards | | Nov 2026 | Jul 2026 | ||
| TritonBench Models write Triton GPU operators in two channels, one from real GitHub kernels (G) and one matching PyTorch operators (T). Execution checks correctness and speedup. | Tsinghua University (THUNLP) | GPU kernels | Expert | Not in cards | | Feb 2025 | Jan 2025 | ||
| UA-Bench Models identify underwater sound sources and their subclasses from audio recordings. The authors repeat the test with silent audio to check reliance on sound. | Zhe Sun and colleagues | Audio understanding and voice | University | Not in cards | | Aug 2026 | Jun 2026 | ||
| UI-Vision Element Grounding Models locate interface elements from instructions. The score is the published overall grounding accuracy. | ServiceNow Research and collaborators | Finding things on screen | Everyday | Not in cards | Claude 3.7 Sonnet 8.27% | Mar 2025 | Feb 2025 | ||
| UniVerseBench Models answer questions about music across more than 38 cultural and linguistic groups. The authors report accuracy by language and skill. | Ziya Zhou and colleagues | Audio understanding and voice | University | Not in cards | | Aug 2026 | Mar 2026 | ||
| VAKRA Agents answer multi-step questions by calling more than 8,000 locally hosted APIs over real databases in 62 domains, along with matching document collections. Tasks need chains of 3 to 7 steps across APIs and documents. A deterministic evaluator replays tools and checks the full trajectory. | IBM Research | Function calling | Everyday | Not in cards | | Mar 2026 | Apr 2026 | ||
| Vals Smith: sashabaranov/go-openai Agents modify sashabaranov/go-openai to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 96.7% | Aug 2026 | Jul 2026 | ||
| Vals Smith: tensorflow/tensorflow Agents modify tensorflow/tensorflow to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Luna 90% | Sep 2026 | Aug 2026 | ||
| Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. | Vals AI | Deep research | Expert | Not in cards | Claude Opus 5.5 57.5% | Jul 2026 | Sep 2026 | ||
| VideoACE Models answer questions about fast actions in short videos. They can inspect selected frames more closely and request hints from a simulated viewer. | KlingAI Research | Video understanding | Everyday | Not in cards | | Nov 2026 | Sep 2026 | ||
| VideoFDB Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes. | NVIDIA | Speech recognition and translation | Everyday | Not in cards | | May 2026 | Mar 2026 | ||
| VideoGameBench Vision-language models play 10 popular 1990s Game Boy and MS-DOS games in real time from raw frames. Scored by the share of each game completed. | Alex Zhang, Ofir Press and collaborators | Video games | Expert | Not in cards | | May 2025 | Apr 2025 | ||
| VideoGameBench Lite Agents play video games while the environment pauses during inference. The headline is the official Overall Score. | Alex Zhang, Ofir Press and collaborators | Video games | Expert | Not in cards | GPT-4o 1.60% | May 2025 | Apr 2025 | ||
| VIF-Bench Models generate images from multiple references and visual instructions. Judges check whether the result follows controls such as pose or lighting without copying unwanted features. | The University of Tokyo | Image generation and editing | Everyday | Not in cards | GPT-Image-1.5 7.60 | Sep 2026 | Sep 2025 | ||
| VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. | Tsinghua University and Zhipu AI (THUDM) | Web browsing | Everyday | Not in cards | GPT-4o 36.2% | Aug 2024 | Jul 2024 | ||
| VTR-Bench Models generate videos containing visible text. The evaluation checks the rendered words and how well the text persists across frames. | Yu Huang and colleagues | Video generation | Everyday | Not in cards | | Oct 2026 | Jul 2026 | ||
| Washington Post political lean evaluation Models answer 30 US political questions in up to 30 words each. A reporter marks left- and right-leaning arguments and counts answers that present both. | The Washington Post | Bias, fairness and politics | Everyday | Not in cards | GPT-5.5 76.7% | Jun 2026 | May 2026 | ||
| WebWalkerQA Agents start from a website's home page and traverse its subpages to find the facts that answer a question. Answers are judged against references. | Alibaba Tongyi Lab | Deep research | Everyday | Not in cards | | Jan 2025 | Sep 2024 | ||
| WorkflowArena Agents write executable workflows that coordinate other agents on professional tasks. Hidden verifiers check the completed deliverables. | Navers Lab, Einsia.AI and Tsinghua University | Multi-agent games | Expert | Not in cards | Claude Opus 5.5 37% | Nov 2026 | Sep 2026 | ||
| WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. | UC Santa Barbara (UCSB NLP) | Embodied reasoning | Everyday | Not in cards | GPT-6 Astra 42.3% | Sep 2026 | Sep 2026 | ||
| X-Stream Models answer questions that combine concurrent video streams. A model judge grades answers, while forward-looking questions also require responses within the correct time window. | Chinese University of Hong Kong MMLab and Huawei | Video understanding | Everyday | Not in cards | | Jun 2026 | Nov 2025 | ||
| XYBench Models answer requests that pursue a mistaken solution to an underlying problem. A model judge checks the useful solution, its emphasis, and identification of the misconception. | Ai2 and collaborators | Overall chat quality | Everyday | Not in cards | GPT-5 31% | Sep 2026 | Apr 2026 | ||
| XYEval Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice. | Google DeepMind | Deception and alignment | Expert | Not in cards | GPT-5.5 60.9% | Sep 2026 | Aug 2026 | ||
| τ^τ-bench Coding agent builds a customer-service agent from business records, a client and an API. The built agent is scored against held-out simulated users. | Sierra + Princeton | Building whole projects | Expert | Not in cards | Claude Opus 5 23.9% | Sep 2026 | Jul 2026 | ||
| Ada-LEval Models put shuffled text segments in order and select the accepted answer from many candidate answers. The benchmark varies input length up to 128,000 tokens and scores outputs against the correct order or answer. | OpenCompass | Reasoning over long documents | Everyday | Not in cards | GPT-4 Turbo 46% | Apr 2024 | Jan 2024 | ||
| AECBench (Liang et al.) Models interpret Chinese building codes, calculate design quantities, and write project documents. Answer keys score objective questions. A model judge grades report generation against expert rubrics. | East China Architectural Design & Research Institute and Tongji University | Expert-written questions | University | Not in cards | | Sep 2025 | Mar 2025 | ||
| Agent4Sci Coding agents try to improve the published best model on 20 scientific ML benchmarks across 11 domains, under long in-the-wild runs. The paper reports mean relative improvement rather than a ranking. | Anthropic | AI research tasks | Research frontier | Not in cards | Claude Opus 4.6 20.1% | Oct 2026 | Mar 2026 | ||
| AgentBoard LLM agents act in 9 partially observable, multi-turn environments that cover embodied tasks, games, web browsing and tool use. Besides final success, AgentBoard reports a progress rate, the share of annotated subgoals the agent reaches. An open toolkit breaks results down by skill. | HKUST NLP and collaborators | General assistants | Everyday | Not in cards | Claude 2 48.9% | Jan 2024 | Sep 2023 | ||
| AgentRewardBench Models judge whether web-agent trajectories completed their tasks. Their judgments are compared with expert labels across 1,302 trajectories from five benchmarks, using precision, recall, and F1. | McGill University, Mila, Google DeepMind, Polytechnique Montreal, and ServiceNow Research | Web browsing | Everyday | Not in cards | o4-mini 82% | Apr 2025 | Apr 2025 | ||
| AI Professions Prompt Benchmark Models draft everyday work documents from fictional inputs: 100 prompts for tasks in 10 professions, such as accounting, law, healthcare administration and real estate, plus Excel and Microsoft 365, in English, Spanish, French and Portuguese. A judge model from another family checks 5 pass or fail criteria per prompt, fixed before any model ran. | AI Professions | Tasks from real jobs | University | Not in cards | Claude Opus 5.5 85.9% | Oct 2026 | Sep 2026 | ||
| AI-Spire (AGI-Eval) Models replay one Slay the Spire 2 seed, build decks, and choose combat moves. The organizers record the floor reached on each attempt. | AGI-Eval Community | Video games | Everyday | Not in cards | Claude Opus 4.7 21.6 | May 2026 | Apr 2026 | ||
| ALCE Models answer questions and cite passages. ALCE evaluates answer quality and checks whether the citations support the answer. | Princeton NLP with human evaluation by Surge AI | Questions that need search | Everyday | Not in cards | | May 2023 | Jul 2023 | ||
| AlephBench Models answer Hebrew reasoning, knowledge, and arithmetic questions. They also extract answers from passages and translate English into Hebrew. | HebArabNlpProject | Reasoning in many languages | School | Not in cards | | Jun 2026 | Apr 2026 | ||
| AlphaZero Connect Four study Coding agents implement an AlphaZero pipeline for Connect Four. The resulting programs play a round-robin tournament against the Pascal Pons solver. | Joshua Sherwood, Ben Aybar, and Benjamin Kaplan | Puzzles and chess | Expert | Not in cards | Claude Opus 4.7 1,938 | Apr 2026 | Apr 2026 | ||
| Amazon Counterfactual Detection Models identify customer reviews that describe an alternative to what actually happened. | James O'Neill and colleagues | Reading in many languages | School | Not in cards | | Apr 2021 | Oct 2026 | ||
| Android Agent Arena (100 tasks) Agents use 20 Android apps to find information and perform actions. A vision model checks whether the agent reached the required intermediate states and completed the task. | Android Agent Arena team | Phones | Everyday | Not in cards | | Aug 2025 | Jun 2025 | ||
| Android Agent Arena (201 tasks) Replaced in Aug 2025 | Not in cards | | Jan 2025 | Jul 2024 | |||||
| AnyAppBench Agents repeat the same task in different Android apps, with and without suggested steps. Each app has a verifier that checks the same success condition. | KAIST and Carnegie Mellon University | Phones | Everyday | Not in cards | | Sep 2026 | Feb 2026 | ||
| Arizona voter questions in English and Spanish Models answer Arizona voter questions in English and Spanish. Fact-checkers flag answers containing a false statement or a broken link. | Proof News, Factchequeado, and the Institute for Advanced Study | Long-form factuality | Everyday | Not in cards | GPT-4 30% | Oct 2024 | Sep 2024 | ||
| ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. | Armnet | Real robots | Everyday | Not in cards | π0.5 47.6% | Jul 2026 | May 2026 | ||
| AudioJevBench Models listen to a clip and assign probabilities to answers about speech or sound. The maker checks accuracy and calibration across 988 scored items. | Benchmark Heaven | Audio understanding and voice | Everyday | Not in cards | | Sep 2026 | Sep 2025 | ||
| AudioTrust Audio language models answer prompts that probe hallucination, robustness, speaker authentication, privacy, fairness, and safety. Automated evaluators, including language-model judges, compare responses with task-specific references and criteria. | Nanyang Technological University, Tsinghua University, Beijing Normal-Hong Kong Baptist University and others | Audio understanding and voice | Everyday | Not in cards | | May 2025 | Sep 2025 | ||
| Bank Question Corpus Models determine whether two Chinese customer service questions ask for the same thing. The authors reserved 10,000 labeled pairs for testing. | Harbin Institute of Technology, Shenzhen, and WeBank | Finance | Everyday | Not in cards | | Sep 2018 | Jun 2025 | ||
| BARC-Bench Agents handle routine workplace requests and use records to infer privacy and safety constraints. A judge checks whether each attempt completes safely or defers. | Scale AI | Deception and alignment | University | Not in cards | GPT-6 Astra 72.7% | Oct 2026 | Sep 2026 | ||
| BEAR Models choose which entity correctly completes a factual relation. The evaluation compares candidate completions across model architectures. | Jacek Wiland and colleagues | Trivia and open questions | School | Not in cards | | Jun 2024 | Feb 2024 | ||
| Beemo Detectors distinguish human writing from model outputs, including texts revised by experts or other models. | Toloka and collaborators | Composite indices | University | Not in cards | | Nov 2024 | Jun 2024 | ||
| Bench4HLS Models write synthesizable C++ from hardware specifications. The grader runs simulation tests, synthesizes each design, and measures resources and latency. | University of Central Florida | Electronics | University | Not in cards | GPT-5 50% | Jan 2026 | Aug 2025 | ||
| Bengali Classical Music Generation Models compose melodies for Bengali lyrics in ABC notation. Three listeners hear rendered samples and judge whether each melody matches the requested style. | University of Central Florida and Auburn University | Creative writing | University | Not in cards | GPT-4o 40% | Jun 2026 | Jun 2025 | ||
| BoardGameBench (Homer Quan) Models play seven board game variants against a search engine. The scorer rewards quick wins and gives partial credit for longer losses. | Homer Quan | Multi-agent games | Everyday | Not in cards | | May 2026 | Aug 2026 | ||
| Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. | Andon Labs | Real robots | Everyday | Not in cards | Claude Opus 4.1 37% | Oct 2025 | Aug 2025 | ||
| CareQA Models answer questions from Spanish specialist healthcare examinations. The evaluation includes English and Spanish versions and an open-response format. | Anna Arias-Duart and colleagues | Medicine | Expert | Not in cards | | Feb 2025 | Jun 2024 | ||
| CFLUE Application Models complete Chinese financial language tasks. The score is the published average of the application task metrics. | Alibaba Cloud (Qwen DianJin) and Soochow University | Finance | University | Not in cards | | May 2024 | Oct 2023 | ||
| ChainWorld Agents complete sequences of desktop tasks while preserving the state needed for later steps. OSWorld checks each step. | Scale AI and UC Berkeley | Desktop apps | Everyday | Not in cards | GPT-5.5 31% | Jun 2026 | Apr 2026 | ||
| CivBench (Civilization VI) Agents play Civilization VI and are evaluated on strategic progress in a long-running world. | CivBench authors | Video games | Expert | Not in cards | | Sep 2026 | Mar 2026 | ||
| CLEF JOKER 2026 Task 4: Humour Generation Models write a joke using a target word in two specified senses. Three language models compare pairs of jokes for each language ranking. | CLEF JOKER organizers | Creative writing | Everyday | Not in cards | Kimi K2.6 1,389 | Oct 2025 | Apr 2026 | ||
| CLEVA Models complete Chinese language tasks including text classification, instruction following, and classical Chinese comprehension. The evaluation samples questions for each round. | Yanyang Li and colleagues | Knowledge in many languages | University | Not in cards | GPT-4 0.96 | Aug 2023 | Jun 2023 | ||
| Climate-Change-NER Models identify and label climate terms in scientific abstracts. The authors provide 13 categories and reserve 75 abstracts for testing. | IBM Research and NASA | Lab and data work | University | Not in cards | GPT-4o 0.44 | Apr 2024 | Jul 2024 | ||
| CodeSearchNet Challenge Models retrieve code snippets using natural language queries. The challenge evaluates how closely ranked results match the search intent. | Hamel Husain and colleagues | Function-level problems | University | Not in cards | | Sep 2019 | Oct 2026 | ||
| CodeTransOcean Models find equivalent code written in different languages or deep learning frameworks. The evaluation compares retrieved snippets with paired implementations. | Weixiang Yan and colleagues | Function-level problems | University | Not in cards | | Dec 2023 | Oct 2026 | ||
| codex of veritas Models answer ledger questions when the supplied quantities determine the result. They must abstain when a necessary quantity is missing. | bahramn | Faithful to sources | School | Not in cards | | Nov 2026 | Sep 2026 | ||
| CompanyBench Agents complete financial analyst workflows in a simulated bank. Checks compare outputs and database changes with expected results. | Turing | Finance | Expert | Not in cards | GPT-5.5 44.4% | May 2026 | |||
| Corpus for Emotion Detection in Russian Models classify Russian sentences by the emotion they express. The reference labels distinguish joy, sadness, surprise, fear, and anger. | Alexander Sboev and colleagues | Reading in many languages | Everyday | Not in cards | | Oct 2026 | |||
| CoSQA Models retrieve code that matches a natural language query. Relevance judgments link the queries with code snippets. | Junjie Huang and colleagues | Function-level problems | University | Not in cards | | Aug 2021 | Oct 2026 | ||
| Coval Speech-to-Text WER Speech models transcribe call-like audio. The score is mean word error percentage over the published pooled 24-hour window. | Coval | Speech recognition and translation | Everyday | Not in cards | | Apr 2026 | Aug 2026 | ||
| CREATE Models find a word that connects three photographs. A judge grades each answer against the intended connection. | Barnard College and Columbia University | General visual questions | Everyday | Not in cards | | Oct 2026 | Aug 2026 | ||
| CroSS-MR Models read a scenario from a novel and select the reason for a character’s decision. The score is the percentage of correct choices. | Fudan University and collaborators | Reading comprehension | School | Not in cards | | Apr 2024 | Feb 2024 | ||
| CTKFacts Models compare Czech claims with evidence and predict whether the evidence supports, refutes, or fails to resolve each claim. | Herbert Ullrich and colleagues | Faithful to sources | School | Not in cards | | Jan 2022 | Oct 2026 | ||
| CUA-World-Long Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task. | Carnegie Mellon University (Welleck and Neubig labs, CMU L3) | Desktop apps | Expert | Not in cards | GPT-5.4 27.5% | Apr 2026 | Mar 2026 | ||
| CURIE Biodiversity Georeferencing Models read a map and its caption from an ecology paper and return the geographic bounds. The score measures overlap with an expert's bounding box. | Google and CURIE collaborators | Spatial reasoning and counting | University | Not in cards | | Feb 2025 | |||
| CURIE Geospatial Dataset Extraction Models read a research paper and list its datasets, variables, sources, time spans, and geographic coverage in JSON. The authors report ROUGE-L and BERTScore. | Google and CURIE collaborators | Finding facts in long inputs | University | Not in cards | | Feb 2025 | |||
| CutVerse Agents complete editing tasks in seven applications, including Premiere Pro, Photoshop, and DaVinci Resolve. Vision models grade progress against milestones in the recorded workflow. | Communication University of China, National University of Singapore, and USEIT AI | Desktop apps | University | Not in cards | Claude Opus 4.6 68.3% | May 2026 | Feb 2026 | ||
| CyberGym-E2E (615 tasks) Agents discover and demonstrate vulnerabilities on the older 615-task cohort. The headline is cumulative stage S3 success. | UC Berkeley RDI (Sunblaze) and collaborators | Finding and exploiting real bugs | Expert | Not in cards | | Jun 2026 | Dec 2025 | ||
| DeepResearch Arena Deep research agents get research tasks generated from academic seminar transcripts, which reflect expert discussion and reduce data leakage. | DeepResearch Arena authors | Deep research | Expert | Not in cards | o4-mini-deep-research 3.96 | Sep 2025 | Jul 2025 | ||
| DeployGuard Models diagnose synthetic deployment logs. The grader checks their JSON answers against known failure stages, exit codes, and operating constraints. | Sasanka Weerakoon | Fixing real repositories | University | Not in cards | GPT-5.4 Mini 0.87 | Oct 2026 | Jul 2026 | ||
| DevBench Acceptance Testing Models generate acceptance tests for repositories. The score is the published oracle-test percentage. | OpenCompass (Shanghai AI Laboratory) and collaborators | Building whole projects | Expert | Not in cards | GPT-4 Turbo (Jan 2024) 29.2% | Mar 2024 | Jan 2024 | ||
| DevBench Environment Setup Models prepare repository environments. The score is the percentage that pass the example usage test. | OpenCompass (Shanghai AI Laboratory) and collaborators | Building whole projects | Expert | Not in cards | GPT-4 Turbo 41.7% | Mar 2024 | Jan 2024 | ||
| DevBench Unit Testing Models generate unit tests for repositories. The score is the published oracle-test percentage. | OpenCompass (Shanghai AI Laboratory) and collaborators | Building whole projects | Expert | Not in cards | GPT-4 Turbo (Jan 2024) 36.5% | Mar 2024 | Jan 2024 | ||
| DEW-LogiQ Models choose answers to 358 questions on agricultural water management. The authors compare ordinary reasoning prompts with prompts that include concept maps. | Michigan State University | Science questions | University | Not in cards | o3 87.4% | Oct 2025 | Jun 2025 | ||
| DiagTrace Models request information about a patient one turn at a time. The evaluator compares the sequence of requests with the published case history. | Yufan Wang and colleagues | Medicine | Expert | Not in cards | Kimi K2.5 0.40 | Nov 2026 | Jan 2026 | ||
| DiscourseEE Models read Reddit discussions about opioid use disorder and extract details about medication use, tapering, and relapse. Some details require inference across a conversation. | Dartmouth College | Medicine | University | Not in cards | | Oct 2024 | Apr 2024 | ||
| DS-1000 Models write Python for 1,000 data science problems from StackOverflow across seven libraries such as NumPy and Pandas. Test cases and API constraints check each answer. | HKU, Stanford, UC Berkeley, UW, Meta and CMU | Data science and SQL | University | Not in cards | GPT-4 Turbo 53.9% | Nov 2022 | Apr 2024 | ||
| EmbodiedCLUE-VLA (planning) Models reason about actions in physical environments. | SuperCLUE | Embodied reasoning | Everyday | Not in cards | GPT-6 Astra 86.8 | Sep 2026 | |||
| ERP-Bench Agents complete 300 procurement and manufacturing workflows in Odoo 19. A constraint solver generates the tasks and optimal solutions, and the verifier checks final business state. | Agentic Labs | Apps and MCP servers | Expert | Not in cards | GPT-5.5 43.4% | May 2026 | Apr 2026 | ||
| ERR Estonian language and cultural history questions Chatbots answer ten questions about Estonian and ten about Estonian cultural history. ERR counts correct answers in each category. | Estonian Public Broadcasting (ERR) | Knowledge in many languages | School | Not in cards | Grok 4 19.0 | Feb 2026 | Dec 2025 | ||
| euchre-bench Coding agents build a euchre bot and test it against four opponents. Each opponent contributes up to 100 wins to the total. | Few word do trick | Building whole projects | University | Not in cards | GPT-5.5 140.0 | May 2026 | May 2026 | ||
| EvoPolicyGym Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves. | EvoPolicyGym authors | AI research tasks | Expert | Not in cards | GPT-5.5 0.89 | Jul 2026 | Jun 2026 | ||
| EXP-Bench Agents get a research question and incomplete starter code from an influential AI paper. They must form hypotheses, design and run experiments, and analyze results. | Curie team (Just-Curieous) | AI research tasks | Research frontier | Not in cards | o3-mini 0.50% | May 2025 | Feb 2025 | ||
| FairScreen: Testing LLM Hiring Decisions Models screen job applicants under a stated policy. Paired cases change identity cues, qualifications, or the completeness of the evidence. | Maksim Silchenko | Bias, fairness and politics | University | Not in cards | | Oct 2026 | Apr 2026 | ||
| Faroese Semantic Textual Similarity Models score how closely two Faroese sentences agree in meaning. The evaluation compares their rankings with reference judgments. | Vésteinn Snæbjarnarson and colleagues | Reading in many languages | School | Not in cards | | May 2023 | Oct 2026 | ||
| Financial Retrieval (Daloopa) Chatbots retrieve historical financial figures. Daloopa compares their answers with normalized reference values. | Daloopa | Finance | University | Not in cards | Claude Opus 4.1 91.6% | Sep 2025 | Aug 2025 | ||
| FinEval 6.0 Models solve financial evaluation tasks. The score is the official 6.0 total score. | Shanghai University of Finance and Economics (AIFLM Lab) | Finance | University | Not in cards | | Aug 2023 | Apr 2025 | ||
| Finnish Parliament ASR test Models transcribe Finnish parliamentary speeches. The original test reports word error rates separately for speakers heard during training and speakers absent from training. | Aalto University | Speech recognition and translation | Everyday | Not in cards | Whisper large-v3 9.29% | Aug 2017 | Oct 2026 | ||
| FollowIR Models rank documents in response to a search instruction. The benchmark tests whether revising the instruction changes the ranking of the affected documents. | Orion Weller and colleagues | Questions that need search | School | Not in cards | | Mar 2024 | Oct 2026 | ||
| FrontierML (Synthetic Sciences) Agents conduct 25 empirical machine learning projects under fixed budgets. Each task has a verifier and reference scores. Harbor includes the 15 public projects. | Synthetic Sciences | AI research tasks | Research frontier | Not in cards | Claude Fable 5.2 0.52 | Oct 2026 | Nov 2026 | ||
| GAIK LA-murre dialect test Models transcribe 583 Finnish dialect interview segments from ten localities. The reference preserves dialect forms, so standard Finnish spellings can count as errors. | GAIK project, Haaga-Helia University of Applied Sciences | Speech recognition and translation | Everyday | Not in cards | | Oct 2026 | May 2026 | ||
| GAIK SKN dialect test Models transcribe 600 Finnish dialect interview segments from ten municipalities. The reference uses standard Finnish word forms. | GAIK project, Haaga-Helia University of Applied Sciences | Speech recognition and translation | Everyday | Not in cards | | Oct 2026 | May 2026 | ||
| GerDaLIR Small Models match German court passages to cited decisions. This version keeps only decisions that answer at least one test query. VAGO scores the top ten results with nDCG@10. | Massive Text Embedding Benchmark team, adapting Marco Wrzalik and Dirk Krechel | Law | University | Not in cards | | Mar 2024 | Oct 2026 | ||
| GerLayQA Models find German Civil Code sections that answer questions from the public. The authors also evaluated generated answers against lawyers' answers. | Marius Büttner and Ivan Habernal | Law | University | Not in cards | | Feb 2024 | Oct 2026 | ||
| GerLeRB Models rank German federal statute sections for 367 legal questions. Legal experts checked the questions against the cited provisions. | University of Jena and collaborators | Law | University | Not in cards | | Jun 2025 | Oct 2026 | ||
| gg-bench Models play 126 games invented by a language model and receive game-specific outcomes. | gg-bench authors | Puzzles and chess | Everyday | Not in cards | o1 36.3% | May 2025 | Feb 2025 | ||
| GoodAI LTM Benchmark Conversational agents retain, update, and use information across conversations with long stretches of intervening text. Task-specific evaluators compare answers with expected responses and assign scores from zero to one. | GoodAI | Reasoning over long documents | Everyday | Not in cards | Claude 3 Opus 8.70 | Feb 2024 | Apr 2024 | ||
| GraphicDesignBench Models analyze and produce graphic designs in 50 tasks spanning layout, typography, infographics, design semantics, and animation. The suite uses professional layered design templates. | LICA World | General visual questions | Expert | Not in cards | GPT-5.4 0.94 | Apr 2026 | Mar 2026 | ||
| GreekBarRetrieval Models retrieve Greek statutory articles for bar exam questions and case facts. The authors compare ranked results with the statutes relevant to each case. | Ernest Beta and colleagues | Law | University | Not in cards | gpt-oss-120b 0.43 | Aug 2026 | Aug 2025 | ||
| Group Binary Search (Riedl) Agents each choose an integer without seeing the other guesses. They adjust after hearing whether the group sum exceeds or falls below a hidden target. | Christoph Riedl (Northeastern University) | Multi-agent games | Everyday | Not in cards | | Jul 2025 | Jul 2025 | ||
| HappyWorld Embodied The HappyWorld arena publishes a separate embodied vote pool and Elo ranking. | Alibaba | Interactive world models | Everyday | Not in cards | | Sep 2026 | |||
| Hardest SAT Problems (Vellum) Models solve 50 hard SAT math problems. A judge compares each final answer with a reference, and a person reviews the result. | Vellum | Word problems | School | Not in cards | Claude Sonnet 4 94% | May 2025 | |||
| HarnessEval-W Video and world models generate rollouts from image and action prompts. An evaluator agent plans checks, spawns sub-agents, gathers spatiotemporal evidence and scores observation, transition and persistence with written reasoning. | MirroS Lab with xbench (HongShan) and universities | Interactive world models | Expert | Not in cards | | Aug 2026 | |||
| HEAD-QA v2 Models answer Spanish healthcare specialization exam questions across six disciplines. Version 2 adds languages, and the MedARC port runs the English configuration. | Alexis Correa-Guillén, Carlos Gómez-Rodríguez, and David Vilares | Medicine | Expert | Not in cards | | Nov 2025 | Jul 2024 | ||
| HMU-Bench Models answer Chinese and English questions about musical notation and recordings. The authors score responses against reference answers. | Zheqi Dai and colleagues | Audio understanding and voice | University | Not in cards | | Oct 2026 | Aug 2026 | ||
| Hungarian Riddles Benchmark Models answer 100 original Hungarian riddles and justify each answer in Hungarian. The author grades factual accuracy, reasoning, and language. | Károly Boczka | Reasoning in many languages | Everyday | Not in cards | GPT-5 67.7% | Oct 2025 | Sep 2025 | ||
| IndicLID benchmark Models identify Indian languages in native scripts and in Romanized text. | Yash Madhani and colleagues | Reading in many languages | School | Not in cards | | May 2023 | Oct 2026 | ||
| ironbench Agents write firmware for tasks such as serial communication and controller sequencing. The grader checks device behavior and counts tasks solved within the allowed attempts. | Tsyren Balmaev | Electronics | University | Not in cards | glm-4.7-flash 69.2% | Sep 2026 | Feb 2026 | ||
| Italian-Administrative-CaseHold Models classify Italian administrative judgments paired with their official holdings by legal subject. | Daniele Licari and colleagues | Law | University | Not in cards | | Oct 2026 | |||
| JSICK Models judge entailment and semantic relatedness in Japanese sentence pairs. An additional stress test examines sensitivity to word order and case marking. | Hitomi Yanaka and Koji Mineshima | Reasoning in many languages | School | Not in cards | | May 2021 | Oct 2026 | ||
| KernelQuest Agents optimize Triton kernels. A task passes when the kernel produces correct outputs and reaches the speed target on the same GPU. | Turing | GPU kernels | Expert | Not in cards | Claude Opus 5.5 79% | Sep 2026 | Sep 2026 | ||
| KeyFrame-Compass Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities. | KlingAI Research (Kuaishou) | Video generation | Everyday | Not in cards | | Jul 2026 | Mar 2026 | ||
| Kinetics-700-2020 Models recognize human actions in short videos from the 2020 edition of Kinetics-700. | DeepMind | Video understanding | Everyday | Not in cards | | Oct 2020 | Jan 2026 | ||
| Kinetics-700 Replaced in Oct 2020 | Not in cards | Jul 2019 | |||||||
| Kinetics-600 Replaced in Jul 2019 | Not in cards | Aug 2018 | |||||||
| Kinetics-400 Replaced in Aug 2018 | Not in cards | | May 2017 | Mar 2025 | |||||
| Korean Sarcasm Dataset Models identify sarcasm in Korean text using labeled examples. | Kocasm project | Reading in many languages | Everyday | Not in cards | | Oct 2026 | |||
| KorSET Models respond to Korean prompts adapted to local language, law, and culture. A model judge checks whether each response violates the relevant safety policy. | DATUMO AI Safety Team | Deception and alignment | Everyday | Not in cards | | Mar 2025 | |||
| Lakh, Crore and Carpet Area Models solve Indian property and finance problems involving local land units, loan payments, and lakh and crore notation. | reet_22 | Word problems | School | Not in cards | Claude Haiku 5.5 97% | Oct 2026 | Oct 2026 | ||
| Legal Consent Sweep Agents review an acquisition data room, reconcile consent requirements, and prepare notices and closing advice. A separate verifier grades their work. | DOE | Law | Expert | Not in cards | Claude Opus 5.5 0.38 | Oct 2026 | Sep 2026 | ||
| LegalQuAD Models find the German legal document that answers each of 200 questions. The original data also mark the answer text and its starting position. | Christoph Hoppe and colleagues | Law | University | Not in cards | | Aug 2021 | Oct 2026 | ||
| LHAW Agents face 285 underspecified variants of long-horizon tasks with goals, constraints, inputs or context removed. Scored on strategic clarification and task success. | Scale AI | General assistants | Expert | Not in cards | GPT-5.2 68.4% | Feb 2026 | Dec 2025 | ||
| LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. | LIBERO-MAX authors | Simulated robots | Everyday | Not in cards | | Aug 2026 | May 2026 | ||
| LLM-IOAA Models solve 49 theory problems and eight data analysis problems from four astronomy olympiads. Two expert graders score the solutions using the official rubrics. | The Ohio State University and collaborators | Science olympiads | Competition | Not in cards | GPT-5 86.3% | Oct 2025 | Aug 2025 | ||
| LogicVal Models generate images from prompts with formal constraints on objects, attributes, counts, and relationships. The evaluator checks which constraints the image satisfies. | Savitha Sam Abraham and colleagues | Image generation and editing | School | Not in cards | | Oct 2026 | |||
| ltzGLUE Models perform language understanding tasks in Luxembourgish, including entailment, grammar judgments, sentiment analysis, and named entity recognition. | Alistair Plum and colleagues | Reasoning in many languages | School | Not in cards | GPT-5 mini 69.4 | Apr 2026 | Aug 2025 | ||
| MafiaBench Models play a social deduction game with hidden roles and public discussion. | MafiaBench | Multi-agent games | Everyday | Not in cards | | Apr 2025 | Feb 2025 | ||
| MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. | Mei Wu and colleagues | Lab and data work | University | Not in cards | GPT-5.4 51.5% | Sep 2026 | Mar 2026 | ||
| MatTools Models answer questions about materials science software and write Python programs to calculate material properties. The authors execute the generated programs. | Siyu Liu and colleagues | Lab and data work | University | Not in cards | | May 2025 | Feb 2025 | ||
| Medbullets Models answer clinical questions patterned on US medical licensing examinations. The benchmark also evaluates explanations of the answers. | Hanjie Chen and colleagues | Medicine | Expert | Not in cards | GPT-4 68.8% | Feb 2024 | Apr 2024 | ||
| MedCalc-Bench (original) Models apply 55 clinical calculators to patient notes. The original test contains 1,100 questions and checks numerical answers against task-specific tolerance bounds. | NIH National Center for Biotechnology Information | Medicine | Expert | Not in cards | | Jun 2024 | Apr 2024 | ||
| MemEye Multimodal agent memory systems answer questions that need fine visual evidence kept from earlier sessions, which captions or text traces alone cannot supply. | MemEye authors (Kwok et al.) | Finding facts in long inputs | Everyday | Not in cards | GPT-5.4 Mini 49.4% | May 2026 | Mar 2026 | ||
| MemoryAgentBench Memory agents receive long inputs in chunks over many turns and are tested on accurate retrieval, test-time learning, long-range understanding and selective forgetting. It adds two new sets, EventQA and FactConsolidation. | UC San Diego (Hu, Wang, McAuley) | Learning from the context | Everyday | Not in cards | GPT-5 mini 60.6% | Jul 2025 | Aug 2025 | ||
| Mercury Models write Python solutions and pass hidden correctness tests. The evaluator compares runtime with reference solutions. Harbor ports all 256 problems from the evaluation split. | Mingzhe Du and colleagues | Speed and optimization | University | Not in cards | | Feb 2024 | Apr 2024 | ||
| MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. | Georgia Institute of Technology | Simulated robots | Everyday | Not in cards | π0.5 50.4% | Nov 2026 | Dec 2025 | ||
| MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. | Google DeepMind | Long videos | Expert | Not in cards | GPT-5 42.2% | Jan 2026 | Sep 2025 | ||
| MisconceptDialogue Tutors identify a simulated student's mathematical misconception and address it through dialogue. The authors compare exact-match and human-calibrated model grading on the same conversations. | Electronics and Telecommunications Research Institute and collaborators | Other professions | University | Not in cards | Claude Haiku 4.5 87.8% | Oct 2026 | Mar 2026 | ||
| MIVAS Bench Voice agents handle calls for a dermatology practice, a law firm, and a retailer. Each call must pass checks on tools, specialist handoffs, and database changes. | Bluejay Labs | Customer service | University | Not in cards | Grok Voice 3.0 56.5% | Aug 2026 | Oct 2026 | ||
| MLRC-Bench Agents propose and implement methods for machine learning competitions. The scorer compares their performance with a supplied baseline and the strongest human solution. | Yunxiang Zhang and colleagues | AI research tasks | Research frontier | Not in cards | | Apr 2025 | Mar 2025 | ||
| MMLU-SR Models answer MMLU questions after key terms are replaced with synonyms. The benchmark tests whether models recognize the same concept under different wording. | Wentian Wang and colleagues | Exam questions | University | Not in cards | GPT-4o mini 65% | Jun 2024 | Jul 2024 | ||
| ModelBench: Reliability Under Constraints Models follow several simultaneous constraints. Paired tasks compare a control case with a harder version, and code checks each response. | Eze Prince jr | Following precise instructions | School | Not in cards | gpt-oss-20b 0.97 | Oct 2026 | Sep 2026 | ||
| MotoVerse Models use a rider's camera to anticipate maneuvers and predict nearby road users' trajectories. The authors compare vision, language, and conventional prediction models. | Varun A. Paturkar and colleagues | Embodied reasoning | Everyday | Not in cards | | Nov 2026 | Oct 2025 | ||
| MTEB-BR Embedding models encode Brazilian Portuguese text for classification, clustering, similarity, retrieval, and reranking tasks. The headline score averages the primary task metrics across 22 native Portuguese tasks. | Tardelli Ronan Coelho Stekel, Federal Institute of Sao Paulo | Reading in many languages | University | Not in cards | | Jul 2026 | Mar 2026 | ||
| MultiEURLEX Models assign subject labels to European Union legislation written in different languages. | Ilias Chalkidis and colleagues | Law | Expert | Not in cards | | Sep 2021 | Oct 2026 | ||
| MusICA-MetaBench Models answer the same music questions from audio, printed scores, and ABC notation. The authors publish fixed tests for two collections of choral music. | Charles University and Brno University of Technology | Audio understanding and voice | University | Not in cards | | Jul 2026 | Mar 2026 | ||
| NFCorpus Models rank medical research documents for health questions expressed in everyday language. | Vera Boteva and colleagues | Medicine | University | Not in cards | | Feb 2016 | Jun 2025 | ||
| OBLIQ-Bench Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems. | MIT (Tchuindjo, Shah and Khattab) | Questions that need search | Expert | Not in cards | GPT-5.2 0.12 | May 2026 | Mar 2026 | ||
| OdysseysMega Agents complete tasks in a web browser. A judge checks each rubric criterion against the recorded actions and screenshots. | Turing | Web browsing | Expert | Not in cards | Claude Opus 5.5 41% | Sep 2026 | |||
| Omni-InteractScope Proactive Models provide timely proactive assistance during audio and video streams. The score is the published T8 assistance recall. | Alibaba Group (Qwen, Alibaba Token Hub) + Hong Kong Polytechnic University | Video understanding | Everyday | Not in cards | | Oct 2026 | May 2026 | ||
| Omnilingua-AudioMC Models follow requirements across spoken conversations in ten languages. The authors compare each model's spoken and text responses against the same 1,404 rubric items. | Nankai University and Alibaba | Speech recognition and translation | Everyday | Not in cards | | Nov 2026 | Sep 2026 | ||
| OpenAudioBench AlpacaEval Models respond to spoken AlpacaEval instructions. The score is the published judge quality score. | Baichuan | Audio understanding and voice | Everyday | Not in cards | | Jan 2025 | Jan 2025 | ||
| OpenAudioBench Llama Questions Models answer spoken Llama Questions. The score is the published accuracy. | Baichuan | Audio understanding and voice | Everyday | Not in cards | | Jan 2025 | Jan 2025 | ||
| OpenAudioBench Reasoning QA Models answer spoken reasoning questions. The score is the published accuracy. | Baichuan | Audio understanding and voice | Everyday | Not in cards | | Jan 2025 | Jan 2025 | ||
| OpenAudioBench TriviaQA Models answer spoken TriviaQA questions. The score is the published judge quality score. | Baichuan | Audio understanding and voice | Everyday | Not in cards | MiniCPM-o-2_6 6.19 | Jan 2025 | Jan 2025 | ||
| OpenAudioBench Web Questions Models answer spoken Web Questions. The score is the published judge quality score. | Baichuan | Audio understanding and voice | Everyday | Not in cards | MiniCPM-o-2_6 6.86 | Jan 2025 | Jan 2025 | ||
| OpenLegalRetrieval-DE Models retrieve German statutes and court decisions across seven tasks. Each task scores where relevant documents appear in the first ten results. Each task carries equal weight. | VAGO Solutions | Law | University | Not in cards | | Nov 2026 | Oct 2026 | ||
| ORAV Bench Models combine reference images, sounds, and videos into a requested audiovisual scene. A judge compares pairs of outputs against each reference's assigned role. | Tsinghua University and Tencent Hunyuan | Video generation | Everyday | Not in cards | | Sep 2026 | |||
| OS-Critic Bench Models judge whether a proposed action advances a task using the screenshot and action history. The grader compares each judgment with a human label. | Shanghai AI Laboratory and collaborators | Finding things on screen | Everyday | Not in cards | GPT-5 68.2% | Dec 2025 | Sep 2025 | ||
| ParallelKernelBench Models replace PyTorch and NCCL reference code with multi-GPU CUDA kernels for 87 problems from codebases such as Megatron-LM and DeepSpeed. Scored on correctness and speed over the baseline. | Together AI and Stanford | GPU kernels | Expert | Not in cards | GPT-5.5 32.2% | Jun 2026 | Jun 2026 | ||
| PazaBench v2 Models transcribe speech in 61 African languages. The leaderboard measures character error rate, word error rate and transcription speed. | Microsoft Research Africa, Nairobi | Speech recognition and translation | Everyday | Not in cards | Whisper large-v3-turbo 73.3% | Jul 2026 | Feb 2025 | ||
| PazaBench v1 Replaced in Jul 2026 | Not in cards | Feb 2026 | |||||||
| PCBWorld-Bench Agents route printed circuit boards in KiCad. A board earns a clean pass when all required connections are complete and KiCad reports zero design-rule errors. | LG AI Research and collaborators | Electronics | University | Not in cards | GPT-5.4 0.54 | Jul 2026 | Mar 2026 | ||
| PII Masking Models locate personal information in US English text and replace it with a common redaction tag. The default split reserves 20% for evaluation. | Adam Lucek | Following precise instructions | Everyday | Not in cards | Claude Sonnet 4.5 0.73 | Oct 2025 | Sep 2025 | ||
| Pipecat Speech-to-Text Benchmark Speech-to-text services transcribe the same 1,000 realtime audio clips. Semantic word error rate measures accuracy, and time to final segment measures streaming latency. | Pipecat | Speech recognition and translation | Everyday | Not in cards | | Aug 2026 | Sep 2026 | ||
| PRDBench Code agents build 50 real-world Python projects across 20 domains from product requirement documents. A trained judge agent checks each project against structured criteria. | AGI-Eval (Meituan) | Building whole projects | Expert | Not in cards | GPT-5.2 69% | Oct 2025 | Dec 2025 | ||
| PRISM-Bench (audio and video generation) Models generate speech, music, or environmental sounds with video. A judge compares outputs with reference clips across 35 criteria, including synchronization and prompt following. | Shanghai Artificial Intelligence Laboratory and Meituan | Video generation | Everyday | Not in cards | | May 2026 | |||
| Project Euler (MathArena) Models solve recent Project Euler problems that need both mathematical insight and code. MathArena adds problems as Project Euler releases them, starting from problem 943. | MathArena (ETH Zurich SRI Lab and INSAIT) | Competition problems with answers | Expert | Not in cards | | Sep 2025 | May 2026 | ||
| PsychEthicsBench Models answer multiple-choice questions and respond to scenarios under Australian psychology and psychiatry ethics guidelines. Graders check responses against professional standards. | Monash University, University of Liverpool, and Hefei University of Technology | Wellbeing and sycophancy | University | Not in cards | | Jan 2026 | Sep 2024 | ||
| PulseBench-Tab Models convert 1,820 table images in nine languages into HTML tables. T-LAG represents adjacent cells as directed edges and matches their text and structure against human annotations. The reported score is F1 over matched edge weights. | Pulse Software Corp | Documents, charts and text in images | Everyday | Not in cards | Claude Fable 5.1 0.89 | Apr 2026 | Sep 2026 | ||
| RadioBench Models answer radio engineering questions and generate programs for software-defined radios. The authors measure successful execution, hardware deployment, and reception of the intended signal. | Jiazhen Lei and the RadioMaster team | Electronics | University | Not in cards | GPT-5.5 97% | Jun 2026 | May 2026 | ||
| Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. | Momentum Research Labs | Simulated robots | Expert | Not in cards | Claude Fable 5.2 0.60 | Nov 2026 | Nov 2026 | ||
| ReasonZoo Models solve tasks in logical reasoning, puzzles, and programming. The evaluation compares their answers with task solutions. | OpenCompass and collaborators | Abstract puzzles and logic | University | Not in cards | | Aug 2025 | May 2025 | ||
| RedlineBench Agent acts as counsel at one turn of a SaaS contract negotiation and returns tracked-change redlines. Three LLM judges grade attorney-written rubrics. | Crosby Intelligence + micro1 | Law | Expert | Not in cards | GPT-5.5 50.5% | Jun 2026 | Jun 2026 | ||
| RefuNQ Models answer factual questions and decide when to decline. The authors create unanswerable questions by replacing a noun in a real question with an invented term. | University of Illinois Urbana-Champaign | Short factual questions | Everyday | Not in cards | GPT-4 53.7% | Nov 2023 | Sep 2023 | ||
| ReviewBench AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1. | GitHub and Microsoft | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 73.3% | Oct 2026 | Sep 2026 | ||
| RoboChallenge Table30 v2 2.0 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. | Dexmal + Hugging Face | Real robots | Everyday | Not in cards | π0.5 0.73% | May 2026 | Apr 2025 | ||
| RoboChallenge 1.0 Replaced in May 2026 | Not in cards | π0.5 42.7% | Oct 2025 | Apr 2025 | |||||
| RoboDx Agents repair faults in robot navigation software. Hidden driving scenarios test the repair. A separate diagnosis track asks models to identify faults without seeing the code. | Soham Dan, Scale AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 100% | Nov 2026 | Oct 2026 | ||
| RSIBench-Data Coding agents act as data-centric researchers. With a fixed Tinker post-training stack and budget, they refine training data from dev-benchmark feedback, and the final model is scored on held-out test benchmarks. | Evolvent AI + National University of Singapore | Machine learning engineering | Research frontier | Not in cards | GPT-5.6 Sol 32.6% | Jul 2026 | Jul 2026 | ||
| ScaLA Models judge whether sentences in mainland Scandinavian languages are linguistically acceptable. | Dan Nielsen | Reading in many languages | School | Not in cards | | May 2023 | Oct 2026 | ||
| SemEval 2022 Task 8 multilingual news article similarity Models rate the similarity of two news articles, including articles in different languages. The evaluation compares predicted similarities with human judgments. | Xi Chen and colleagues | Reading in many languages | School | Not in cards | | Jul 2022 | Jun 2025 | ||
| SemEval 2024 Task 1 semantic textual relatedness Models rank sentence pairs by relatedness across languages. The evaluation compares their rankings with human judgments. | Nedjma Ousidhoum and colleagues | Reading in many languages | School | Not in cards | | Mar 2024 | Oct 2026 | ||
| Skill Reward Hacking Researchers track when models learn to increase a proxy reward without improving task quality. They measure warning time, false alarms, and transfer across reward functions. | tonyteo | Deception and alignment | University | Not in cards | | May 2026 | Apr 2026 | ||
| Small Model Agent Bench Models choose tools and supply their arguments in 28 Chinese and English episodes. Fixed tool responses let the grader check planning, recovery, and task completion. | Cacinie | Function calling | School | Not in cards | | Aug 2026 | Sep 2026 | ||
| SmartPlay Language-model agents play a collection of games designed to test planning, reasoning, memory and interaction abilities. | Microsoft Research | Puzzles and chess | Expert | Not in cards | GPT-4 0.80 | Oct 2023 | Jul 2023 | ||
| Snorkel Working Memory Test Models answer questions from a document placed among distractors. The evaluation varies the document position and the length of the input. | Snorkel AI | Finding facts in long inputs | School | Not in cards | GPT-4 Turbo 87.6% | Jul 2024 | Apr 2024 | ||
| SQL Reality Check Models write SQL for revenue and ticket histories. The grader runs each query on seven databases with cases such as partial refunds and tied timestamps. | Dhruv Bansal | Data science and SQL | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| Stack EOL Audit: geol CLI vs API Agents identify software components that have reached end of life. They audit the same twelve stacks from memory, with an API, and with a command line tool. | Adrien Sales | Function calling | University | Not in cards | | Oct 2026 | Apr 2026 | ||
| SuperCLUE-DSPSafeBench Models respond to Chinese prompts that test content safety. | SuperCLUE | Deception and alignment | Everyday | Not in cards | | Sep 2024 | |||
| SuperCLUE-TTS Models read Chinese text aloud. The benchmark rates basic speech synthesis and delivery in situations such as public announcements. | SuperCLUE | Audio understanding and voice | Everyday | Not in cards | | Dec 2024 | Sep 2026 | ||
| SuperCLUE-VLR Models solve problems that require visual reasoning. | SuperCLUE | General visual questions | University | Not in cards | | May 2025 | |||
| SuperCLUE-World World models generate visual sequences. | SuperCLUE | Video world models | University | Not in cards | | ||||
| SWE-Pair Agents fix two real bugs from the same Python project using one combined issue report. Hidden tests from both pull requests grade each fix. | Evaligo Labs | Fixing real repositories | Expert | Not in cards | | Nov 2026 | Sep 2026 | ||
| T2I-NSI Models generate images from paired positive and negative prompts. The index measures whether removing a requested concept also removes it from the image. | Indian Institute of Technology Jodhpur and Colorado State University | Image generation and editing | Everyday | Not in cards | GPT-Image-1.5 0.1 | Nov 2026 | |||
| T2LSC-Bench Models render text at a specified location in an image. Judges check the lettering and whether its meaning alters other objects or the background. | Yan Wang and colleagues | Image generation and editing | Everyday | Not in cards | | Sep 2026 | Feb 2026 | ||
| TaskBench Models decompose requests, choose tools and dependencies, and predict tool parameters. Text metrics score decomposition, while graph and parameter F1 scores measure the remaining stages. | Zhejiang University, Microsoft Research Asia and Fudan University | Function calling | Everyday | Not in cards | Claude 2 93.5 | Nov 2023 | Jan 2024 | ||
| TavernBench Agents build a Hearthstone Battlegrounds player through research, simulator updates, and live games. | TavernBench authors | Video games | Expert | Not in cards | GPT-6 Astra 95.6 | Oct 2026 | Sep 2026 | ||
| TeluguMixBench Models answer Telugu and mixed Telugu and English prompts. The harness reports accuracy across comprehension, reasoning, translation, and instruction following. | Poojitha Saranya | Reasoning in many languages | School | Not in cards | Claude Haiku 4.5 100% | Oct 2026 | Oct 2026 | ||
| TextEdit Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity. | OpenCompass (Shanghai AI Laboratory) | Documents, charts and text in images | Everyday | Not in cards | | Mar 2026 | |||
| ToolQA Models answer questions that need outside information, using 13 tools such as database queries, text retrieval and a calculator over fixed data sources. The questions are built so that a model's internal knowledge cannot answer them. An easy and a hard set separate single lookups from multi-step tool use, and answers are checked by exact match. | Georgia Institute of Technology | Function calling | Everyday | Not in cards | davinci-003 24.1% | Jun 2023 | Jul 2023 | ||
| UniREditBench Image editing models carry out 2,700 edits that need reasoning, in real-world scenes and in game worlds with human-defined rules, across 8 primary and 18 secondary dimensions. A multimodal judge scores each edit against both a text reference and a ground-truth image. | Fudan University and collaborators | Image generation and editing | Everyday | Not in cards | | Nov 2025 | |||
| Vals Smith: arandomhooman/hoomans-morphe-patches Agents modify arandomhooman/hoomans-morphe-patches to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 20% | Aug 2026 | Jul 2026 | ||
| Vals Smith: chrisanderton/tyk-gateway Agents modify chrisanderton/tyk-gateway to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 76.7% | Aug 2026 | Jul 2026 | ||
| Vals Smith: Expertium/fsrs-autoresearch Agents modify Expertium/fsrs-autoresearch to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Sonnet 5 96.7% | Sep 2026 | Jul 2026 | ||
| Vals Smith: geopandas/geopandas Agents modify geopandas/geopandas to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | | Aug 2026 | Jul 2026 | ||
| Vals Smith: goharbor/harbor Agents modify goharbor/harbor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 60% | Aug 2026 | Jul 2026 | ||
| Vals Smith: ladydascalie/core Agents modify ladydascalie/core to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | | Aug 2026 | Jul 2026 | ||
| Vals Smith: leanprover-community/mathlib4 Agents modify leanprover-community/mathlib4 to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 76.7% | Aug 2026 | Aug 2026 | ||
| Vals Smith: skypilot-org/skypilot Agents modify skypilot-org/skypilot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 4.8 30% | Aug 2026 | Jul 2026 | ||
| Vals Smith: suitedaces/dorabot Agents modify suitedaces/dorabot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | | Aug 2026 | Aug 2026 | ||
| VDEBench Image editing models make local text and table edits to 3,506 Chinese and English document images. Scores check the edit and that the rest of the page is preserved. | Yi et al. (VDE-Bench authors) | Documents, charts and text in images | Everyday | Not in cards | FLUX.2 [klein] 4B 0.77 | Jan 2026 | Sep 2025 | ||
| VELOCITI Models distinguish a video's correct description from a caption that assigns an action to the wrong person. | Darshana Saravanan and colleagues | Video understanding | School | Not in cards | LLaVA-OneVision 72B 43.1% | Jun 2024 | Aug 2024 | ||
| VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. | Alibaba Group, Shanghai Jiao Tong University, and Fudan University | Video generation | Everyday | Not in cards | | Sep 2026 | |||
| VocalGrad Models decide which way an acoustic property changes within a recording. The authors compare model accuracy with human listeners on the same binary questions. | The University of Tokyo and Tohoku University | Audio understanding and voice | Everyday | Not in cards | Kimi-Audio-7B-Instruct 51.8% | Oct 2026 | Dec 2025 | ||
| VSI-Bench-Debiased Models answer the 2,362 VSI-Bench questions left after removing 2,768 that text-only models could solve from answer priors or world knowledge. | New York University (VisionX) | Spatial reasoning and counting | Everyday | Not in cards | | Nov 2025 | Apr 2025 | ||
| WebHarbor Agents navigate local copies of 153 websites to search, sign in, book services, and change stored state. The suite contains 3,518 tasks, including 306 verified test tasks and 2,288 training tasks. The test score measures the percentage of tasks passed by scripted verifiers. | AIMING Lab | Web browsing | Everyday | Not in cards | GPT-6 Astra 86.7% | May 2026 | Sep 2026 | ||
| WeGenEditBench Models edit images from instructions that can include text or multiple references. A shared checklist scores each requested change and the details that should stay intact. | Tencent WeChat Vision and collaborators | Image generation and editing | Everyday | Not in cards | | Oct 2026 | |||
| WorkArena++ Web agents complete 682 compositional knowledge-work tasks in ServiceNow, which need planning, reasoning and memory across many steps. | ServiceNow Research | Web browsing | Expert | Not in cards | GPT-4o 1.90% | Jul 2024 | May 2024 | ||
| WorkSurface-Bench Agents choose documents, tables, or graphs to answer workplace questions. The benchmark scores where agents look, what evidence they find, and their answers. | Hao Liang and colleagues | Documents and spreadsheets | University | Not in cards | GPT-5.5 10.3 | Jul 2026 | Apr 2026 | ||
| WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. | WorldArena team (Tsinghua FIB Lab) | Interactive world models | Not in cards | | May 2026 | Sep 2025 | |||
| WorldArena 1.0 Replaced in May 2026 | Tsinghua FIB Lab | Video world models | Not in cards | | Feb 2026 | ||||
| WorldExam Controllable video generators are tested as world models. Levels run from visual appearance to whether the depicted world reacts plausibly beyond what the prompt states. | WorldExam authors | Video world models | Everyday | Not in cards | | Aug 2026 | |||
| Writing Benchmark (Every) Models complete eight writing and editing assignments. Every compares drafts with published writing and checks whether edits match its editors’ choices. | Every | Taste and style | University | Not in cards | Claude Opus 4.8 67.8 | Jun 2026 | Jun 2026 | ||
| YoCausal Video generation models are tested for causal and arrow-of-time understanding by comparing real videos with their time-reversed versions, in two levels. | Alaya Lab | Physical prediction | Everyday | Not in cards | | May 2026 | Jul 2025 | ||
| Act2Answer Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control. | Act2Answer authors (CognitiveAISystems) | Simulated robots | Everyday | Not in cards | | Jun 2026 | Apr 2026 | ||
| ActiveVision Models inspect images repeatedly to count regions, trace structures, and compare visual attributes. The released split has five examples for each of 17 tasks. Exact match checks the final answer after normalizing case, spacing, and separators. | University of Southern California | Spatial reasoning and counting | Everyday | Not in cards | Claude Fable 5 50.6% | Jul 2026 | Jun 2026 | ||
| AgentEscapeBench Agents solve procedural escape rooms using unfamiliar tools. The evaluator checks whether they complete dependent steps and escape. | AGI-Eval (Meituan) | Puzzles and chess | Everyday | Not in cards | Claude Opus 4.6 76.9% | May 2026 | Mar 2026 | ||
| AgenticDataBench Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios. | Tsinghua University | Data science and SQL | Expert | Not in cards | Kimi K2.5 48.8% | Jul 2026 | Feb 2026 | ||
| AgenticSTS Agents play the Silent in Slay the Spire 2 at its lowest difficulty. The authors compare wins with different memory settings and two other agents. | Xiangchen Cheng and colleagues | Video games | Everyday | Not in cards | | Jun 2026 | Apr 2026 | ||
| AI Detection Benchmark Systems label private pre-2022 human documents and AI rewrites of them as human or AI written. Both LLMs and commercial detectors are scored by balanced accuracy. | Vals AI | Taste and style | Expert | Not in cards | Claude Opus 5.5 98.4% | Sep 2026 | Sep 2026 | ||
| AnomalyXL Models locate and describe anomalies in long synthetic time series. The benchmark checks multiple-choice answers and scores precise answers using the known anomaly properties. | Nicolas Zumarraga and the TimeRLM team | Reasoning over long documents | Expert | Not in cards | GPT-5.5 0.33 | Aug 2026 | Apr 2026 | ||
| APE (Attempt to Persuade Eval) A model is asked to persuade a simulated user across 600 topics, from benign facts to conspiracies and clearly harmful ones. A judge scores whether it attempts persuasion. | FAR.AI | Refusing harmful requests | Everyday | Not in cards | o4-mini 10% | Jun 2025 | May 2025 | ||
| Arctic-Extract Business Document Benchmark Models answer questions about business documents and extract information from varied layouts. The internal visual tasks include contracts, signatures and checkboxes. | Snowflake | Documents, charts and text in images | University | Not in cards | GPT-5 58.5 | Aug 2025 | Aug 2025 | ||
| ArenaWebdevCorrectness Judges inspect generated applications and predict whether their features work. Human labels provide the reference answers. | Arena | Web and app front ends | University | Not in cards | GPT-5.6 Sol 81.6% | Oct 2026 | Jul 2026 | ||
| Argus-Bench v2 Models screen inputs to a language model gateway. The benchmark checks detection rates, false positives, and agreement with the policy on severity. | Karmendra Pandey | Jailbreaks and prompt injection | University | Not in cards | | Nov 2026 | Sep 2026 | ||
| ASAG2024 Models grade short answers from seven English datasets on a common zero to one scale. The authors weight errors to account for rare grades. | Zurich University of Applied Sciences | Other professions | University | Not in cards | GPT-4o 0.27 | Jun 2024 | May 2024 | ||
| AssemblyAI Speech Benchmarks Speech recognition systems transcribe clean, accented, multilingual and code-switched audio, identify entities and diarize speakers. Error rates compare production systems across public speech sets. | AssemblyAI | Speech recognition and translation | Expert | Not in cards | | Nov 2026 | Aug 2026 | ||
| AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. | Google DeepMind and ETH Zurich | Machine learning engineering | Expert | Not in cards | Claude 3.7 Sonnet 22% | Mar 2025 | Feb 2025 | ||
| AutoBench (HDL testbench generation) Models write self-checking Verilog testbenches. The evaluator checks compilation, acceptance of correct designs, and agreement with reference testbenches on faulty designs. | Ruidi Qiu and colleagues | Electronics | University | Not in cards | Claude 3.5 Sonnet 71.1% | Jul 2024 | Aug 2024 | ||
| Automated Student Assessment Prize 2.0 Models score source-based essays from seven prompts on a six-point scale. The competition measures agreement with human grades using quadratic weighted kappa. | The Learning Agency Lab and Vanderbilt University | Other professions | University | Not in cards | | Apr 2024 | Oct 2024 | ||
| Automated Student Assessment Prize: Automated Essay Scoring v1 Replaced in Apr 2024 | William and Flora Hewlett Foundation and Kaggle | Not in cards | Monkey 0.00 | Jan 2012 | Nov 2023 | ||||
| AutoPatchBench Agents write security patches for crashes that a fuzzer found in native code. Fixes are checked by fuzzing and differential debugging. | Meta (PurpleLlama) | Secure code and review | Expert | Not in cards | | Apr 2025 | Apr 2025 | ||
| B-MoCA Agents use Android apps such as Instagram on phones with different languages, font sizes, and icon positions. Task-specific checks decide whether each attempt succeeded. | KAIST and Yonsei University | Phones | Everyday | Not in cards | GPT-4o 47.7% | Apr 2024 | May 2024 | ||
| Base64Bench Models encode and decode base64 strings across different text and data formats. The evaluator measures exact matches and edit similarity. | Rich Barton-Cooper | Following precise instructions | School | Not in cards | Claude 3.7 Sonnet 75.9% | Oct 2025 | Aug 2025 | ||
| BigEarthNet.txt Models answer land cover questions, caption satellite images, and locate regions described in text. The authors checked annotations for 1,082 pairs of radar and optical images. | BIFOLD and Technical University of Berlin with collaborators | General visual questions | University | Not in cards | GLM-4.6V-Flash 0.79% | Mar 2026 | Dec 2025 | ||
| BRIGHT-Pro Models retrieve passages that cover different parts of a question. The 739 questions include 115 from Earth Science Stack Exchange. | Yale NLP and collaborators | Questions that need search | University | Not in cards | | Apr 2026 | Jun 2025 | ||
| BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. | Xiangyu Xu and colleagues | Real robots | Everyday | Not in cards | π0-FAST 0.56 | Oct 2026 | Apr 2025 | ||
| CADExpert Models write CadQuery programs from dimensioned engineering drawings. The graders check whether each program runs and compare its 3D shape with the reference. | Ke Niu and colleagues | CAD models | University | Not in cards | | Dec 2025 | Aug 2025 | ||
| Cekura Speech to Text Overall word error rate across the published audio clips. | Cekura | Audio understanding and voice | Everyday | Not in cards | GPT-Realtime-Whisper 2.63% | Jul 2026 | May 2026 | ||
| Cents Matter Models solve Brazilian Portuguese accounting cases in matched pairs. The grader checks amounts to the cent across rounding, cash flow signs, and cases with insufficient evidence. | Bruno Fernandes | Finance | University | Not in cards | | Oct 2026 | Aug 2026 | ||
| ChemPro Models answer chemistry questions across four difficulty levels. The authors report multiple-choice accuracy and accept numerical answers within specified tolerances. | Aaditya Baranwal and colleagues | Science questions | University | Not in cards | | Feb 2026 | Apr 2025 | ||
| ChouBun Models answer questions about long Japanese documents and summarize company securities reports. | Sakana AI | Reasoning over long documents | University | Not in cards | | Oct 2024 | Apr 2024 | ||
| CLEF JOKER 2026 Task 3: Onomastic Wordplay Translation Models translate English names containing wordplay into French using a short description. The official score accepts reference matches and alternatives approved by translators. | CLEF JOKER organizers | Translation | University | Not in cards | | Oct 2025 | Dec 2025 | ||
| CLIMATE-FEVER Models find evidence for climate claims and decide whether it supports or refutes each claim. | Thomas Diggelmann and colleagues | Faithful to sources | University | Not in cards | | Dec 2020 | Jun 2025 | ||
| CMI-RewardBench Models judge musical quality and how well recordings follow requests. Some requests combine text, lyrics, and reference audio. Scores measure agreement with human listeners. | Queen Mary University of London, Peking University, and collaborators | Audio understanding and voice | Everyday | Not in cards | | Feb 2026 | Nov 2025 | ||
| CodeContests Models write programs for competitive programming problems, drawn from Codeforces and other sites, and the programs run against hidden tests. Google DeepMind built the dataset for AlphaCode, with training, validation (117 problems) and test (165 problems) splits and many generated tests per problem to cut false positives. Scores are the share of problems solved within a fixed number of submissions, such as 10@k. | Google DeepMind | Competitive programming | Competition | Not in cards | GPT-5.4 81.8% | Feb 2022 | Apr 2026 | ||
| CommonGround Elicit Models identify ambiguities, contradictions and gaps in organizational documents. They select a clarification and predict how each stakeholder will respond. | Charlie Thompson | Reading comprehension | University | Not in cards | GPT-4.1 0.40 | Aug 2026 | Sep 2025 | ||
| CompMat-Bench Agents complete individual steps from published computational materials studies. Fixed rules compare their inputs and outputs with results reproduced by the authors. | Chenmu Zhang and colleagues | Lab and data work | Expert | Not in cards | GPT-5.6 Sol 91.5% | Sep 2026 | Sep 2026 | ||
| CoSE-E Models transcribe business conversations that switch between languages. The main metric checks whether transcription errors change the answers to questions about the conversation. | ServiceNow AI Research | Speech recognition and translation | Everyday | Not in cards | | Sep 2026 | Dec 2025 | ||
| Cost Per Correct v2 Models answer tasks at three difficulty levels and repeat some with paraphrased instructions. Rules check correctness so the author can compare cost per correct answer. | Karmendra Pandey | General assistants | University | Not in cards | gpt-oss-120b $91 | Nov 2026 | Sep 2026 | ||
| Cost Per Correct v1 Replaced in Nov 2026 | Not in cards | Oct 2026 | |||||||
| CRAB Multimodal agents do tasks that span an Ubuntu desktop and an Android phone at once. Graph-based evaluators give credit for intermediate steps. | CAMEL-AI and collaborators | Desktop apps | Everyday | Not in cards | GPT-4o 38% | Jul 2024 | May 2024 | ||
| CultureAtlas Models judge whether statements about a region or cultural group are true. The authors pair statements drawn from Wikipedia with false alternatives and report F1. | University of Illinois at Urbana-Champaign | Commonsense | Everyday | Not in cards | | Feb 2024 | Jul 2023 | ||
| DCASE 2025 Audio Question Answering Models answer questions about marine mammal calls, the timing of sounds, and overlapping events. Each of the three sections counts equally toward the final score. | DCASE 2025 Task 5 organizers | Audio understanding and voice | University | Not in cards | Kimi-Audio-7B-Instruct 52.1% | Dec 2024 | Apr 2025 | ||
| Decision Index Decision models read a state and a typed question and return a decision. The index weights private same-skill sets and new-domain sets, equated on 21 base models. | Hugging Face community (apolinario, multimodalart) | Composite indices | Everyday | Not in cards | | Sep 2026 | Aug 2026 | ||
| DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design. | OpenCompass (Shanghai AI Laboratory) and collaborators | Building whole projects | Expert | Not in cards | GPT-4 Turbo 41.7% | Mar 2024 | Jan 2024 | ||
| DevBench Implementation Models implement repository code. The score is the acceptance-test pass rate, with unit-test pass rate as an auxiliary metric. | OpenCompass (Shanghai AI Laboratory) and collaborators | Building whole projects | Expert | Not in cards | GPT-4 Turbo (Jan 2024) 7.10% | Mar 2024 | Jan 2024 | ||
| DI-Bench Agents answer questions using company tables, documents, and business rules. The benchmark checks whether agents apply those rules when computing results. | Jiangyun Zhang and colleagues | Running a business | University | Not in cards | Claude Opus 4.6 57.9% | Sep 2026 | Feb 2026 | ||
| DolphinBench Agents use long-term memory to complete 600 tool-using tasks for three simulated knowledge workers with roughly 500,000 user-message tokens each. | Mem0 | General assistants | Expert | Not in cards | GPT-5.6 Luna 70.7% | Sep 2026 | Jul 2026 | ||
| Domain Dynamics English-Hindi Translation Models translate 2,094 English sentences into Hindi across six domains. The authors report BLEU and COMET on the full set and by domain. | Indian Institutes of Technology Patna and Jodhpur | Translation | University | Not in cards | | Aug 2024 | Aug 2024 | ||
| DriveBench Behavior Models answer driving behavior questions. The score is the published GPT evaluation score for this task family. | UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU | Embodied reasoning | Everyday | Not in cards | | Jan 2025 | Aug 2024 | ||
| DriveBench Perception Models answer driving perception questions. The score is the published GPT evaluation score for this task family. | UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU | Embodied reasoning | Everyday | Not in cards | Qwen2-VL-7B-Instruct 35.2 | Jan 2025 | Aug 2024 | ||
| DriveBench Planning Models answer driving planning questions. The score is the published GPT evaluation score for this task family. | UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU | Embodied reasoning | Everyday | Not in cards | Qwen2-VL-7B-Instruct 57.0 | Jan 2025 | Aug 2024 | ||
| DriveBench Prediction Models answer driving prediction questions. The score is the published GPT evaluation score for this task family. | UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU | Embodied reasoning | Everyday | Not in cards | | Jan 2025 | Aug 2024 | ||
| DSCodeBench Models write Python functions for 1,000 data science problems covering ten libraries. The evaluator checks generated code against reference outputs. | Shuyin Ouyang and collaborators | Data science and SQL | University | Not in cards | | May 2025 | Nov 2024 | ||
| Duplicate Side Effect Desk Agents process refund requests when timeouts or duplicate tickets obscure payment status. The evaluator uses the transaction ledger to identify duplicate refunds. | jigonyoo | Customer service | Everyday | Not in cards | Claude Sonnet 4.5 0.98 | Sep 2026 | Oct 2025 | ||
| DyCodeEval Models solve dynamically rewritten HumanEval and MBPP problems with equivalent program behavior. The evaluator runs tests and computes DyPass@K across generated variants. | Columbia University | Function-level problems | School | Not in cards | | Mar 2025 | Sep 2024 | ||
| EduVideoBench Models generate classroom videos intended to teach facts, skills, or attitudes. Experts assess instructional accuracy, presentation, and educational safety. | Korea University and collaborators | Video generation | University | Not in cards | | May 2026 | |||
| EgoHome Models annotate 137 egocentric clips of household chores, about two hours in all, with subtasks, hand actions, hand captions and hand boxes. Segments are as short as 0.1 seconds. A model judge matches segment labels, and boxes are scored by centroid hits. | Perceptron | Video understanding | Everyday | Not in cards | GPT-6 Astra 0.80 | Nov 2026 | Sep 2026 | ||
| EHRNoteQA Models read a patient's accumulated MIMIC-IV discharge summaries and answer clinical questions about them. The set has 962 question and answer pairs, each for a different patient. | KAIST and collaborators (Kweon et al.) | Medicine | Expert | Not in cards | GPT-4 96.2% | Feb 2024 | Dec 2023 | ||
| Eleusis Agents play cards to discover a hidden rule. The scorer tests whether a proposed rule makes the same decisions as the hidden rule. | np_hard (nph4rd) | Multi-agent games | Competition | Not in cards | GPT-6 Sol 82.2% | Jun 2026 | Sep 2026 | ||
| EvalCheck-LLM Models use supplied evidence to answer a question, abstain, or flag a conflict. The task tests whether pressure changes that decision. | Akash_ | Faithful to sources | School | Not in cards | | Oct 2026 | Jul 2026 | ||
| EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. | Rocktim Jyoti Das and colleagues | Knowledge in many languages | School | Not in cards | | Mar 2024 | Jan 2024 | ||
| FarSick Models rate how closely two Persian sentences agree in meaning. The original corpus also labels whether one sentence entails or contradicts the other. | Zahra Ghasemi and Mohammad Ali Keyvanrad | Reading in many languages | Everyday | Not in cards | | Sep 2021 | Jun 2025 | ||
| Fechamento BR Models normalize fields in fictional Brazilian administrative records. Paired cases test how context changes the interpretation of money, missing values, dates, and identifiers. | gabriel fernandes | Documents and spreadsheets | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| FieldWorkArena V2.0 Agents inspect workplace images, videos, manuals, and logs to plan actions and report operational events. The evaluator compares their answers and actions with reference results. It combines correctness judgments with partial credit for numerical agreement. | Fujitsu and Carnegie Mellon University | Tasks from real jobs | University | Not in cards | GPT-5.2 52% | Jun 2025 | Dec 2025 | ||
| Fin-Fact Models judge whether evidence supports a financial claim, contradicts it, or leaves too little information. The original evaluation also tests models with images. | Illinois Institute of Technology and China University of Petroleum | Finance | University | Not in cards | Claude 3 Opus 61% | Sep 2023 | Feb 2024 | ||
| First Proof Third Batch Batch 3 The editors run five AI systems once each on ten unpublished research problems with proofs of up to 12 pages. About 30 referees score correctness, exposition and attribution. The score is average correctness, from negative 1 to 1. | First Proof Foundation | Research mathematics | Research frontier | Not in cards | GPT-6 Astra 0.65 | Oct 2026 | Sep 2026 | ||
| First Proof Second Batch Batch 2 Replaced in Oct 2026 | Not in cards | GPT-5.5 Pro 5.00 | Jun 2026 | Apr 2026 | |||||
| First Proof Batch 1 Replaced in Jun 2026 | First Proof (Abouzaid, Blumberg, Hairer, Spielman, Williams and others) | Not in cards | Feb 2026 | ||||||
| FlashInfer-Bench Agents generate GPU kernels for LLM inference that are benchmarked on workloads from real serving traces and can be deployed into the FlashInfer stack. | FlashInfer team | GPU kernels | Expert | Not in cards | GPT-5 0.47× | Jun 2025 | Aug 2025 | ||
| FML-bench Agents propose and run experiments on eight machine learning research tasks. The authors measure task performance and how widely the proposals vary across iterations. | Qiran Zou and colleagues | AI research tasks | Expert | Not in cards | Claude Opus 4.1 0.50 | Oct 2025 | Aug 2025 | ||
| FollowBench Models answer 820 English and Chinese instructions where each level adds one more constraint of content, situation, style, format or example type. Rules and an LLM judge check each constraint. | HKUST and Huawei Noah's Ark Lab (Jiang et al.) | Following precise instructions | Everyday | Not in cards | GPT-4 Turbo 73.4% | Oct 2023 | Nov 2023 | ||
| FVEval Models write SystemVerilog assertions from properties or circuit designs. A formal verifier checks syntax and tests whether assertions express the reference behavior. | NVIDIA and collaborators | Electronics | Expert | Not in cards | | Jun 2024 | Jul 2024 | ||
| GAOKAO-Eval 2024 Models take Chinese national college entrance examinations. High school teachers grade the open responses. | OpenCompass and collaborators | Exam questions | School | Not in cards | | Jun 2024 | Jun 2024 | ||
| GenPuzzle Models generate images that solve visual puzzles such as mazes, tangrams, and Sudoku. The authors select an automated judge using human reference scores. | Changpeng Zhao and colleagues | Image generation and editing | School | Not in cards | | Sep 2026 | Feb 2026 | ||
| GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. | Jinghe Wang and colleagues | Embodied reasoning | Everyday | Not in cards | | Nov 2026 | Apr 2025 | ||
| Grid Perception Reliability Models inspect text grids to count connected shapes, choose valid cells, and translate positions. The harness checks responses against generated ground truth. | Flamki | Spatial reasoning and counting | School | Not in cards | | Oct 2026 | Aug 2026 | ||
| GSR-Bench Models turn speech into transcripts, meeting notes, or other requested text. Judges check whether the output preserves the meaning and stays grounded in the recording. | China Telecom and Northwestern Polytechnical University | Speech recognition and translation | Everyday | Not in cards | | Oct 2026 | Feb 2026 | ||
| GUI Odyssey Mobile GUI agents carry out tasks that cross several apps, such as sharing a post from one social app to another. The dataset has 8,334 episodes averaging 15.3 steps, over 212 apps, 1,357 app combinations and six devices. Agents are scored by how often each predicted action matches the recorded step. | OpenGVLab (Shanghai AI Laboratory) and collaborators | Phones | Everyday | Not in cards | GPT-4V 55% | Jun 2024 | Jul 2024 | ||
| HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. | Seoul National University (SNU PI) | Real robots | Everyday | Not in cards | | Oct 2026 | Jul 2026 | ||
| ImgMT Models translate text embedded in diagrams, maps, and infographics. Evaluation measures text recognition, region matching, and translation with CER, F1, chrF, and COMET. | Šárka Uramová, Petar Petrov, Josef Jon, Ondřej Bojar, and Michal Novák | Translation | Everyday | Not in cards | Kimi K3 61.3 | Oct 2026 | Jul 2026 | ||
| IMO-GradingBench Models grade 1,000 olympiad proof attempts, and their grades are compared with human expert grades. Tests automatic evaluation of long-form math answers. | Google DeepMind (Superhuman Reasoning team) | Written proofs | Competition | Not in cards | o3 54% | Nov 2025 | Aug 2025 | ||
| IndicContextEval Models transcribe speech in eight Indian languages using hints about the topic or names. One prompt deliberately supplies names from an unrelated field. | AI4Bharat at IIT Madras and Sarvam AI | Speech recognition and translation | Everyday | Not in cards | | Jun 2026 | Dec 2025 | ||
| IndicIPR-QA Models extract answers from passages in seven Indian intellectual property laws. The benchmark compares predicted spans with reference answers across 4,462 questions. | South Asian University | Law | University | Not in cards | | Oct 2026 | Sep 2024 | ||
| InfiHall-Bench Models generate images from 411 prompts that combine visual requirements. The evaluator checks details such as object parts, actions, and spatial relations separately. | Harbin Institute of Technology, Shenzhen and collaborators | Image generation and editing | Everyday | Not in cards | FLUX.1 [dev] 67.1 | Nov 2026 | Jan 2025 | ||
| InstrumentalEval Models respond to scenarios where gaining resources or preserving themselves could serve another goal. A grader identifies responses that pursue those intermediate goals. | Yufei He and colleagues | Deception and alignment | Everyday | Not in cards | o1-mini 43.4% | Feb 2025 | Jan 2025 | ||
| IntPhys2-Timing Models identify when a physical rule breaks in a video. The authors add violation intervals to IntPhys2 and score temporal localization. | Taekoan Yoo and colleagues | Physical prediction | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| KnowMe-Bench Models read long autobiographical narratives and must infer a person's stable motivations and decision principles, not just retrieve facts. | QuantaAlpha | Learning from the context | Everyday | Not in cards | GPT-5 mini 50.2 | Jan 2026 | Nov 2025 | ||
| Labelbox Agentic Search Agents search for answers to 200 questions across several knowledge domains. Labelbox reports overall and domain scores. | Labelbox | Questions that need search | University | Not in cards | | Jun 2025 | Jun 2025 | ||
| Labelbox Deep Research Agents Agents investigate research questions at the doctoral level. Labelbox compares their answers and reports scores by domain. | Labelbox | Deep research | Expert | Not in cards | | Jul 2025 | Jun 2025 | ||
| LAMBADA Multilingual (Stable LM 2) Models predict missing final words in narrative passages translated for the Stable LM 2 evaluation. The harness provides five language configurations. | Stability AI | Reading in many languages | Everyday | Not in cards | | Feb 2024 | Feb 2024 | ||
| LHRS-Bench Models answer 690 single-choice questions about 108 remote sensing images. The questions cover land use, object locations, image resolution, and other properties. | Nanjing University | General visual questions | University | Not in cards | Claude 3 Opus 39.1% | Mar 2024 | Feb 2024 | ||
| LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. | Fudan University + Tongji University + Shanghai Innovation Institute + NUS | Simulated robots | Everyday | Not in cards | | Oct 2025 | Feb 2025 | ||
| LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. | Huazhong University of Science and Technology + Harvard + MIT + others | Simulated robots | Everyday | Not in cards | π0.5 53% | Oct 2025 | Apr 2025 | ||
| LLM Game Benchmark Models compete in tic-tac-toe, Connect Four, Gomoku, and other grid games through a shared interface. | LLM Game Benchmark authors | Puzzles and chess | Everyday | Not in cards | Claude 3.5 Sonnet 185.0 | Jan 2024 | Jun 2024 | ||
| LongExtractionBench Extraction systems pull structured values from 225 long table-heavy documents averaging 358 pages and about 88,700 ground-truth fields each. | micro1 | Documents, charts and text in images | Expert | Not in cards | Claude Opus 4.8 70.7% | Jun 2026 | May 2026 | ||
| MarsRetrieval Models match Mars images to captions, retrieve examples of named landforms, and locate features in a global mosaic of the planet. | Southern University of Science and Technology | Spatial reasoning and counting | University | Not in cards | | Feb 2026 | Oct 2026 | ||
| MEDEC Models identify and correct errors in clinical notes across five categories. The OpenMed port uses the MEDEC-MS subset. | Asma Ben Abacha and colleagues | Medicine | Expert | Not in cards | o1-mini 69.1% | Dec 2024 | Feb 2025 | ||
| MedLFQA Models write long answers to biomedical questions. The benchmark checks whether answers include required facts and avoid claims identified as incorrect. | Minbyul Jeong and colleagues | Medicine | Expert | Not in cards | Claude 3 Sonnet 77.1 | May 2024 | Feb 2024 | ||
| Mind2Web-SC Models generate guardrails that check a web agent’s proposed actions against user constraints. The benchmark measures whether the guardrail permits or blocks each action correctly. | Zhen Xiang and colleagues | Web browsing | University | Not in cards | | Jun 2024 | Dec 2024 | ||
| MU-Bench Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts. | Sierra | Speech recognition and translation | Everyday | Not in cards | | Apr 2026 | Aug 2026 | ||
| Multi Agent Coordination Bench Teams of eight agents work for four hours on twelve tasks built so each agent depends on its teammates. Each model also runs alone and as a team that can't talk, which isolates the gain from coordination. | Bespoke Labs | Multi-agent games | Expert | Not in cards | Claude Opus 5.5 0.53 | Nov 2026 | Sep 2026 | ||
| Neural Activation Reading for Collusion Benchmark Monitors estimate whether committee members secretly coordinated their decisions. The transcript task measures how well probability estimates distinguish matched collusion and control cases. | Aaron Rose and colleagues | Deception and alignment | University | Not in cards | gpt-oss-20b 0.87 | Apr 2026 | Aug 2025 | ||
| neuralCAD-Edit Agents edit existing CAD models from text or recorded instructions. CAD experts rate whether they would accept each edit. Software also compares the resulting geometry. | Autodesk Research | CAD models | University | Not in cards | GPT-5.2 0.25 | Apr 2026 | Dec 2025 | ||
| NitiBench Models retrieve Thai statutes and answer legal questions. The authors score retrieval separately and use a model judge to check coverage and contradictions. | Pawitsapak Akarajaradwong and colleagues | Law | University | Not in cards | Claude 3.5 Sonnet 70.3 | Feb 2025 | Sep 2024 | ||
| NormAd-Eti Models decide whether behavior in an everyday story is socially acceptable in the stated country. Questions vary how much cultural guidance the model receives. | Akhila Yerukola and colleagues | Reasoning in many languages | Everyday | Not in cards | | Apr 2024 | Feb 2024 | ||
| OOLONG-Pairs Models identify pairs of users whose records satisfy a condition across a long input. The evaluation compares the predicted pairs with the reference set. | Alex L. Zhang and colleagues | Reasoning over long documents | University | Not in cards | Claude Opus 5 92.9% | Dec 2025 | Jul 2026 | ||
| OpenEarthAgent Agents use tools such as QGIS to measure features in satellite images and maps. The authors grade tool decisions and final answers. | Mohamed bin Zayed University of Artificial Intelligence and collaborators | Apps and MCP servers | University | Not in cards | GPT-5 44.5% | Feb 2026 | Aug 2025 | ||
| OpinionQA Models answer public opinion survey questions. The evaluation compares model response distributions with those of demographic groups. | Shibani Santurkar and colleagues | Bias, fairness and politics | Everyday | Not in cards | GPT-3 175B (davinci) 0.79 | Mar 2023 | Nov 2022 | ||
| Pakistan Local Context LLM Suite Models answer questions about Pakistani geography, local time, currency notation, and Urdu usage. | Tooba Munir | Knowledge in many languages | School | Not in cards | | Oct 2026 | Sep 2026 | ||
| People Making Conversation Speech models continue real conversations at recorded pauses and turn endings. The benchmark compares their timing, response length, and recall with the original speakers. | Arya Bastani and colleagues | Speech recognition and translation | Everyday | Not in cards | | Oct 2026 | Mar 2026 | ||
| Per-DOC Models use a reader’s previous comparison to predict which of two new plots the reader prefers on each of five criteria. | Carnegie Mellon University, Meta AI and UC Berkeley | Taste and style | Everyday | Not in cards | | Oct 2023 | Jul 2023 | ||
| Per-MPST Models use a reviewer’s earlier movie ratings and reviews to predict their rating of a new plot. The authors measure correlation with human ratings. | Carnegie Mellon University, Meta AI and UC Berkeley | Taste and style | Everyday | Not in cards | | Oct 2023 | Jul 2023 | ||
| PhyRubric-Bench Models generate videos of physical events. A judge checks caption-specific requirements against evidence at particular times in each clip. | University of Illinois Chicago and collaborators | Video generation | School | Not in cards | | Nov 2026 | Jul 2025 | ||
| PlanBench Models construct action sequences that reach a goal under stated planning rules. A planning validator checks whether each plan satisfies action preconditions and achieves the goal. | Arizona State University and Colorado State University | Abstract puzzles and logic | School | Not in cards | | Jun 2022 | Jan 2025 | ||
| PRMBench Models identify errors and redundant steps in mathematical solutions. PRMScore combines positive and negative F1 scores across simplicity, soundness, and sensitivity tests. | Fudan University, Shanghai AI Laboratory, Soochow University, Stony Brook University, and Chinese University of Hong Kong | Word problems | University | Not in cards | o1-mini 68.8 | Jan 2025 | Feb 2025 | ||
| Q-Bench-Video Models judge the quality of natural, AI-generated and computer-graphics videos through yes-or-no, what-how, open-ended and pairwise comparison questions. | Q-Future (Nanyang Technological University and Shanghai Jiao Tong University) | Video understanding | Everyday | Not in cards | GPT-4o mini 52.2% | Sep 2024 | Jul 2024 | ||
| Q2D-Web Retrievers search 190 million web documents for queries that agents reformulate from real conversations. | Perplexity | Questions that need search | University | Not in cards | | Sep 2026 | Jun 2025 | ||
| Qualcomm Interactive Cooking Models watch a cook follow a recipe. They give timed instructions, recognize completed steps, and warn about mistakes. The benchmark scores completion accuracy and mistake detection. | Qualcomm AI Research and the University of British Columbia | Video understanding | Everyday | Not in cards | | Nov 2025 | Sep 2025 | ||
| Qwen-Audio-Agent Smart Cockpit Benchmark Agents operate simulated vehicle, navigation, music, and weather services. The short suite checks whole cases. The long suite checks tool use on each turn. | Alibaba | Function calling | Everyday | Not in cards | | Sep 2026 | Aug 2026 | ||
| RAFT (Real-world Annotated Few-shot Tasks) Models learn text classification tasks from small labeled samples. The tasks use real documents, including customer messages, court decisions, and scientific abstracts. | Neel Alex and colleagues | Reading comprehension | University | Not in cards | GPT-3 175B (davinci) 62.7% | Sep 2021 | Mar 2021 | ||
| RealBench (Verilog) Models write Verilog modules or complete systems from design specifications. Testbenches check behavior, and a formal checker compares the result with the reference implementation. | Institute of Computing Technology, Chinese Academy of Sciences, and collaborators | Electronics | Expert | Not in cards | o1-preview 6.65% | Jul 2025 | Sep 2024 | ||
| RealCompanion Models read months of real chats between ten people and an AI companion, then answer questions about who each person is. The release has 27,218 messages over up to 120 days. | Quis Lab | Multi-turn conversation | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| Reasoning Dial Models solve deduction, arithmetic, and distractor tasks at different reasoning settings. The evaluation records accuracy, output tokens, and cost. | Abeera Lodhi | Language reasoning | School | Not in cards | Claude Haiku 5.5 99.6% | Oct 2026 | Oct 2026 | ||
| Red-Pen Bench Models review code with planted defects and fixed or clean controls. The grader checks whether they identify the defect and locate the affected lines. | Robert Gilbert USA | Fixing real repositories | University | Not in cards | Claude Haiku 5.5 0.64 | Oct 2026 | Oct 2026 | ||
| Redtape Models assess eligibility for public benefits and identify when a missing fact prevents a decision. PolicyEngine supplies the reference determinations. | jakpotvin | Law | Expert | Not in cards | GPT-5.6 Sol 0.79 | Sep 2026 | Sep 2026 | ||
| RepoNormBench Agents change code in existing repositories while following the contribution rules that apply to each task. | SYSUSELab | Fixing real repositories | Expert | Not in cards | | Oct 2026 | Apr 2026 | ||
| ReportBench Agents write literature surveys. The evaluator compares their references with published surveys and checks whether cited sources support their claims. | ByteDance BandAI | Deep research | Expert | Not in cards | OpenAI deep research 0.39 | Aug 2025 | May 2025 | ||
| Resolve Models edit sixteen functions. Hidden tests check the requested behavior and verify that previously working behavior still passes. | Aditya Kumar Puri | Function-level problems | University | Not in cards | Claude Haiku 4.5 87.5% | Oct 2026 | Apr 2026 | ||
| ReTrace Computer agents recover from injected errors in desktop tasks. Replaying the remaining actions checks whether the recovered state produces the original successful outcome. | KAIST, ETRI, and DeepAuto.ai | Desktop apps | Everyday | Not in cards | GPT-5.4 63.6% | Oct 2026 | Mar 2026 | ||
| RevBench (Polyvor Labs) Agents analyze supplied artifacts in 22 local reverse engineering puzzles. The verifier checks each submitted answer for an exact match. | Polyvor Labs | Capture-the-flag challenges | Expert | Not in cards | GPT-5.5 50% | Aug 2026 | Aug 2026 | ||
| RMS-AQA Models identify sound events in domestic spaces, then answer questions about their positions and timing. The benchmark scores both stages. | Peihao Chen and colleagues | Audio understanding and voice | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. | Robocurve | Real robots | Everyday | Not in cards | GPT-6 Astra 52.5% | Sep 2026 | Sep 2026 | ||
| RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. | Carnegie Mellon University (Welleck lab, CMU L3) | Simulated robots | Expert | Not in cards | Claude Opus 5.5 0.06 | Oct 2026 | Oct 2026 | ||
| RSCC Models describe disaster damage from satellite images taken before and after an event. The authors reserve 988 image pairs from 19 events for testing. | Zhejiang University | Detection and captioning | University | Not in cards | | Apr 2025 | Apr 2025 | ||
| SeedBench (Seed Science) Models answer questions, summarize passages and interpret research on rice breeding in Chinese and English. The 11 task types use accuracy, macro F1 or ROUGE-L. | Shanghai Artificial Intelligence Laboratory and collaborators | Science questions | University | Not in cards | | Dec 2024 | Mar 2025 | ||
| SMM4H-HeaRD 2026 Detection of Insomnia in Clinical Notes Models identify likely insomnia in nursing and discharge notes. They label diagnostic criteria and give the character offsets of supporting evidence. | Cedars-Sinai and collaborators | Medicine | University | Not in cards | Nov 2025 | ||||
| SMM4H-HeaRD 2025 Detection of Insomnia in Clinical Notes Replaced in Nov 2025 | Not in cards | o3-mini 0.97 | Feb 2025 | Jan 2025 | |||||
| Spyfall study Models play Spyfall while inferring hidden locations and managing deceptive dialogue. | Spyfall study authors | Multi-agent games | Everyday | Not in cards | | Jan 2026 | Apr 2025 | ||
| Summary of a Haystack Models read a synthetic haystack of conversation or news documents where specific insights repeat, then write a query-focused summary that cites its sources. An automatic evaluation scores insight coverage and citation accuracy. | Salesforce AI Research | Reasoning over long documents | University | Not in cards | Claude 3 Opus 42.5% | Jul 2024 | Mar 2024 | ||
| SuperCLUE-ASR Models transcribe Chinese speech under different acoustic conditions. | SuperCLUE | Speech recognition and translation | Everyday | Not in cards | GPT-Transcribe 12.1% | Aug 2026 | |||
| SuperCLUE-Auto Models answer Chinese automotive questions through multiple turns. A judging model scores each answer against criteria for its task. | SuperCLUE team | Other professions | University | Not in cards | Baichuan2-13B-Chat 76.6 | Dec 2023 | Oct 2023 | ||
| SuperCLUE-ICabin Models respond to requests for an automotive cabin assistant. | SuperCLUE | General assistants | Everyday | Not in cards | GPT-4o 82.2 | Aug 2024 | |||
| SuperCLUE-Open Models answer Chinese open-domain questions over multiple turns. The authors compare models across skill categories. | SuperCLUE team | Chat and instructions in many languages | Everyday | Not in cards | GPT-4 94.6 | Jul 2023 | Jan 2024 | ||
| SuperCLUE-VideoEdit Models edit videos in response to Chinese instructions. | SuperCLUE | Video generation | University | Not in cards | | ||||
| SWE-Race Agents fix concurrency and asynchronous lifecycle bugs in Python projects. The benchmark has 95 public and 93 private tasks. Harbor distributes 94 public tasks. | Evaligo Labs | Fixing real repositories | Expert | Not in cards | GLM-5.3-Flash 81.9% | Oct 2026 | Aug 2026 | ||
| TCS-Bench Models write proofs of target results from STOC, FOCS and SODA papers, given the needed context. A verification agent, checked against expert judgments, grades each proof. | Google Research + collaborators (Cohen-Addad, Woodruff and others) | Research mathematics | Research frontier | Not in cards | | Aug 2026 | Aug 2026 | ||
| TELBench Models identify harmful error spans in 1,000 deep-research agent trajectories. Expert labels distinguish mistakes from harmless exploration and tool noise. Evaluation reports macro precision, recall, F1, and first-error accuracy. | Nanjing University NJU-LINK and JIUTIAN Research | Faithful to sources | University | Not in cards | Claude Sonnet 4.6 54.9% | Jun 2026 | Mar 2026 | ||
| Terminal-Bench 2.0 Verified (Z.ai) Agents do the Terminal-Bench 2.0 tasks after Z.ai fixed environments for 89 tasks and instructions for 11. A further 2026-08-18 round fixed tests and instructions in 6 more tasks. | Z.ai (Zhipu AI) | Terminal tasks | Expert | Not in cards | Claude Opus 4.5 61.8% | Feb 2026 | Feb 2026 | ||
| Terminal-Bench Data Agents solve terminal tasks in machine learning, data analysis, and scientific computing. Task verifiers grade the outputs. | Turing | Terminal tasks | Expert | Not in cards | GPT-5.5 48.6% | Jul 2026 | May 2026 | ||
| TextEdit Real Models edit text in real images in the full TextEdit test set. The score is the published judged average. | OpenCompass (Shanghai AI Laboratory) | Documents, charts and text in images | Everyday | Not in cards | GPT-Image-1.5 0.90 | Mar 2026 | |||
| TextEdit Virtual Models edit text in virtual images in the full TextEdit test set. The score is the published judged average. | OpenCompass (Shanghai AI Laboratory) | Documents, charts and text in images | Everyday | Not in cards | GPT-Image-1.5 0.90 | Mar 2026 | |||
| The Amazing Agent Race Agents navigate websites, call tools, and calculate a passcode across 1,400 scavenger hunts. The benchmark includes 800 sequential puzzles and 600 with branching paths. | Minnesota NLP | Function calling | Expert | Not in cards | Claude Sonnet 4 36.6% | Apr 2026 | Mar 2026 | ||
| Tricky Edits of Collected and Curated Images Models edit a source image to follow a written instruction. People and a model judge instruction following, preservation of unaffected content, and visual quality. | Google DeepMind | Image generation and editing | Everyday | Not in cards | | May 2026 | Feb 2026 | ||
| USACO Models solve 307 USA Computing Olympiad problems. Hidden unit tests check correctness and efficiency, scored as pass@1. | Princeton University | Competitive programming | Competition | Not in cards | GPT-4 Turbo 20.2% | Apr 2024 | Feb 2024 | ||
| USIM underwater robot tasks Agents follow instructions to navigate, grasp, transport, and track objects underwater. The simulator measures task completion across 700 trials. | Institute of Automation, Chinese Academy of Sciences, and collaborators | Simulated robots | Everyday | Not in cards | π0.5 37.6% | Oct 2025 | Jun 2025 | ||
| VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. | Yang Qu and colleagues | Embodied reasoning | Everyday | Not in cards | | Apr 2026 | Apr 2025 | ||
| Vals Inference Provider Index Agents use different inference providers to complete the same finance, coding, and legal tasks. | Vals AI | Composite work indices | Expert | Not in cards | | Nov 2026 | Sep 2026 | ||
| Vals Smith: arduino/arduino-ide Agents modify arduino/arduino-ide to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Luna 43.3% | Aug 2026 | Jul 2026 | ||
| Vals Smith: AumeshMisra/tensorzero Agents modify AumeshMisra/tensorzero to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Luna 86.7% | Oct 2026 | Jul 2026 | ||
| Vals Smith: conjfrnk/blackbird Agents modify conjfrnk/blackbird to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 56.7% | Aug 2026 | Aug 2026 | ||
| Vals Smith: dwojtaszek/zipper Agents modify dwojtaszek/zipper to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 93.3% | Sep 2026 | Jul 2026 | ||
| Vals Smith: fmtlib/fmt Agents modify fmtlib/fmt to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Grok 4.5 83.3% | Sep 2026 | Aug 2026 | ||
| Vals Smith: NVIDIA/nccl Agents modify NVIDIA/nccl to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 46.7% | Aug 2026 | Aug 2026 | ||
| Vals Smith: OpenTTD/OpenTTD Agents modify OpenTTD/OpenTTD to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 100% | Sep 2026 | Jul 2026 | ||
| Vals Smith: punt-labs/lux Agents modify punt-labs/lux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 73.3% | Sep 2026 | Aug 2026 | ||
| Vals Smith: vastsa/PI-Desktop Agents modify vastsa/PI-Desktop to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | | Sep 2026 | Aug 2026 | ||
| VAmoS Pro Bench Voice agents handle several utility billing requests in one call. A model checks the spoken amounts and recorded actions against the requirements for each request. | Veris AI | Customer service | University | Not in cards | | Sep 2026 | Sep 2026 | ||
| WebCanvas Web agents do tasks on live websites, scored on key intermediate states rather than one fixed action path, so the test survives site changes. | iMean AI | Web browsing | Everyday | Not in cards | GPT-4o 28.8% | Jun 2024 | Jul 2024 | ||
| WebLINX Agents follow a user's multi-turn chat instructions to navigate real websites. 100,000 interactions from 2,300 expert demonstrations give turn-level metrics. | McGill University and Mila | Web browsing | Everyday | Not in cards | GPT-4V 10.4 | Feb 2024 | Nov 2023 | ||
| Wiki Live Challenge Deep research agents write articles on topics of 100 recent Wikipedia Good Articles, which serve as expert references. Scored on 39 writing criteria and factual verifiability. | USTC and Metastone Technology (DeepResearch Bench team) | Deep research | Expert | Not in cards | GPT-5 53.6 | Feb 2026 | Aug 2025 | ||
| WorkBuddy User Blind Evaluation Agents complete everyday work requests. WorkBuddy users rate their satisfaction on a four-point scale without seeing the model identity. | Tencent | Documents and spreadsheets | University | Not in cards | | Nov 2026 | Oct 2026 | ||
| WorldState-Bench Video models generate two clips from the same image and action. A hidden state changes which ending the rule requires. A vision model checks both clips. | Tsinghua University and GigaAI | Video world models | Everyday | Not in cards | | Oct 2026 | Jul 2025 | ||
| WRENCH (Docker Compose) Agents maintain a service pipeline through seeded disruptions. The grader measures retained throughput and recovery. | WRENCH authors | Terminal tasks | University | Not in cards | Claude Sonnet 5 0.81 | Sep 2026 | Jun 2026 | ||
| Addis AI Amharic Text to Speech Benchmark Speech models read 100 Amharic prompts aloud. Native speakers rate naturalness, intelligibility and voice quality without seeing the system names. | Addis AI | Audio understanding and voice | Everyday | Not in cards | | Aug 2026 | Apr 2026 | ||
| AerialSim Agents fly a simulated drone using camera observations or write a program for the flight. The simulator checks the recorded trajectory against the mission requirements. | Nahid Alam | Simulated robots | Expert | Not in cards | Claude Haiku 4.5 73.2% | May 2026 | Feb 2026 | ||
| AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. | Stanford and Johns Hopkins (Schmidgall et al.) | Medicine | Expert | Not in cards | GPT-4 37.8% | May 2024 | May 2024 | ||
| Almieyar Models transcribe newly recorded speech in 17 Arabic dialects. The authors report word and character error rates across six dialect families. | Omid Ghahroodi and colleagues | Speech recognition and translation | Everyday | Not in cards | gpt-4o-transcribe 34.9% | Sep 2026 | Mar 2025 | ||
| AMBER Multimodal models are checked for object, attribute and relation hallucinations in generative and discriminative tasks, scored without an LLM judge. | Wang et al. (AMBER authors) | Hallucination rates | Everyday | Not in cards | | Nov 2023 | Nov 2023 | ||
| AMELIA Models classify passages from Italian tax decisions as premises or conclusions. They distinguish factual from legal premises and identify the reasoning behind legal premises. | University of Bologna and collaborators | Law | University | Not in cards | | Oct 2024 | Jul 2024 | ||
| AndroidArena Agents use apps such as Gmail and Google Maps, sometimes switching apps or following extra constraints. GPT-4 judges success from the recorded actions and observations. | Peking University and Microsoft Research | Phones | Everyday | Not in cards | | Feb 2024 | Jul 2023 | ||
| App-Bench Coding agents build six full-stack web applications from a single prompt. Two developers grade implemented features against 151 rubric items. The score uses each tool’s best result from three attempts per application. | AfterQuery | Building whole projects | University | Not in cards | Claude Opus 4.5 67.5% | Dec 2025 | Nov 2025 | ||
| ArabCulture-Dialogue Models pick the most culturally fitting reply in dialogues from 13 Arab countries, in Modern Standard Arabic and each local dialect. It also has translation and dialect-steering tasks. | MBZUAI (Al Kautsar, Almheiri et al.) | Chat and instructions in many languages | Everyday | Not in cards | GPT-5 95% | Apr 2026 | Aug 2025 | ||
| ArenaWebdevPreferences Judges compare pairs of generated applications and predict which application the user preferred. The score measures agreement with user votes. | Arena | Web and app front ends | Everyday | Not in cards | | Oct 2026 | Jul 2026 | ||
| Aspire Agent gets only a vague capability goal and must choose data, training and self-checks to evolve weights or harness. Scored on 520 hidden items. | ByteDance Seed et al. | AI research tasks | Research frontier | Not in cards | | Aug 2026 | Feb 2026 | ||
| ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. | BAAI FlagEval | Real robots | Everyday | Not in cards | π0.5 48.2% | Jun 2026 | Dec 2025 | ||
| Audio Arithmetic: 24 Music Workflow Checks Models calculate frame counts, storage, timing, and resampling values for music workflows. The grader compares integer answers with exact reference calculations. | wenye li | Word problems | University | Not in cards | gpt-oss-20b 100% | Oct 2026 | Oct 2025 | ||
| AutoCFD Incompressible Flow Benchmark Agents configure OpenFOAM simulations from flow descriptions and meshes. The authors check solver convergence and compare simulated flow with reference values. | Zhehao Dong, Zhen Lu, and Yue Yang | Scientific code | University | Not in cards | | Apr 2025 | Dec 2024 | ||
| AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. | UC Berkeley | Real robots | Everyday | Not in cards | | Mar 2025 | Jun 2024 | ||
| Automated LLM Speedrunning Benchmark Agents try to reproduce 19 past records of the NanoGPT speedrun, given the earlier code and hints of varying detail. Scored by how much of each record's speedup they recover. | Meta | AI research tasks | Research frontier | Not in cards | o3-mini 46% | Jun 2025 | Feb 2025 | ||
| AutoPLC Models write Structured Text for CODESYS and Siemens controllers from requirements. The benchmark measures how often generated programs compile and how many errors the compiler reports. | Beihang University | Electronics | University | Not in cards | | Dec 2024 | Sep 2024 | ||
| BIG-bench English Proverbs Models read a short story and choose the English proverb that fits it. The 34 stories offer three to six choices each. | Daniel Moseguí González | Reading comprehension | Everyday | Not in cards | | Jan 2021 | Dec 2021 | ||
| BIG-bench Understanding Fables Models read a paraphrased Aesop fable and choose its moral from five options. The test contains 189 fables. | Denis Emelin | Reading comprehension | Everyday | Not in cards | | Jun 2021 | Dec 2021 | ||
| Bimanual Chore Scheduling Models schedule household chores for a simulated two-arm robot. The grader checks resource constraints and compares completion time against an exact optimum. | jwilksbooth | Embodied reasoning | University | Not in cards | Claude Haiku 4.5 0.82 | Jul 2026 | May 2026 | ||
| BlenderBench (Refresh) Agents edit Blender scenes across 20 tasks. Each task grades the submitted scene using its own scoring configuration. | Refresh | CAD models | University | Not in cards | GPT-6 Astra 0.69 | Sep 2026 | Sep 2026 | ||
| Blicket Test Causal Reasoning Models experiment with a simulated detector to infer which objects activate it. The hub version rewards correct identification and informative experiments. | Irfan Jamil | Abstract puzzles and logic | Everyday | Not in cards | | Feb 2026 | Apr 2025 | ||
| BoundaryBench: Policy Under Pressure Models apply a refund policy to twelve cases. The grader compares decisions across clean messages, harmless distractions, and instructions that conflict with the policy. | Arya Koste | Customer service | Everyday | Not in cards | GPT-5.4 Nano 100% | Oct 2026 | Mar 2026 | ||
| BuiltBench Models match a building component description to the corresponding entry in a construction classification system. | Mehrzad Shahinmoghadam and colleagues | CAD models | Expert | Not in cards | | Nov 2024 | Jun 2025 | ||
| CaptionSmith Models convert word-timed transcripts into SRT subtitles. The grader checks text preservation, segmentation, timing, line length and reading speed. | thesatvik | Following precise instructions | University | Not in cards | | Aug 2026 | Apr 2026 | ||
| ClothDojo Robot policies use both hands to flatten and fold garments. The simulator checks success on familiar garments and garments withheld from training. | South China University of Technology and collaborators | Simulated robots | Everyday | Not in cards | π0.5 44.7% | Oct 2026 | Apr 2026 | ||
| Code Interpreter Benchmark General Models use a Python interpreter for general tasks. The score is the percentage of generated programs that execute. | Qwen | Function calling | School | Not in cards | | Sep 2023 | Oct 2023 | ||
| CodePDE Models write a Python solver for each of five families of partial differential equations. Harbor tests each solver on 100 instances and checks its normalized error. | Shanda Li and colleagues | Scientific code | Expert | Not in cards | o4-mini 38.1% | May 2025 | Oct 2025 | ||
| cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability. | Cognition | Building whole projects | Expert | Not in cards | o1-preview 51.8% | Sep 2024 | Sep 2024 | ||
| Contextual Earnings-22 Models transcribe clips from earnings calls with a custom vocabulary. The benchmark measures keyword recognition and transcription errors. | Argmax | Speech recognition and translation | Everyday | Not in cards | | Mar 2026 | Jan 2026 | ||
| ContinualSkillBench Agents work through five domains of 100 linked subtasks of rising difficulty, testing whether skill libraries they build in context actually help later tasks. | Peking University | Learning from the context | Expert | Not in cards | GPT-5.3 Codex 64.1% | Aug 2026 | Apr 2026 | ||
| CoST English-Hindi Models translate 1,947 English sentences into Hindi across eight genres, including plays, poetry and conversation. The organizers report BLEU alongside reference-based and reference-free translation metrics. | MT-NLP Lab, IIIT Hyderabad | Translation | University | Not in cards | | Jun 2024 | Aug 2024 | ||
| Coval Text-to-Speech TTFA Speech models synthesize customer-service prompts. The score is median time to first audible sample in milliseconds over the published 24-hour window. | Coval | Speech recognition and translation | Everyday | Not in cards | Grok TTS 267.3 | Apr 2026 | Apr 2026 | ||
| CoVR-R Models find the video that matches a reference clip after applying a change described in text. | Omkar Thawakar and colleagues | Video understanding | School | Not in cards | | Mar 2026 | Jan 2025 | ||
| CPU Inference Optimization and Streaming Agents optimize llama2.c inference on a CPU. The harness checks numerical output, generated text, and streaming latency across repeated executions. | 朝闻道 | Speed and optimization | Expert | Not in cards | Claude Sonnet 5.5 3.69 | Oct 2026 | Sep 2026 | ||
| CRAG Retrieval-augmented systems answer 4,409 factual questions across finance, sports, music, movies and open domain, using mock web and knowledge graph search APIs. Hallucinated answers are penalized. | Meta | Faithful to sources | Everyday | Not in cards | GPT-4 Turbo 0.20 | Jun 2024 | Apr 2024 | ||
| CTIBench Models answer cyber threat intelligence questions. Tasks cover knowledge questions, CVE to CWE mapping, CVSS severity scoring, threat actor attribution and technique extraction. | Rochester Institute of Technology (Alam, Bhusal, Nguyen, Rastogi) | Finding and exploiting real bugs | Expert | Not in cards | | Jun 2024 | Apr 2024 | ||
| CyberSecEval MITRE Models are asked to help with attack steps drawn from the MITRE ATT&CK framework. Scored by how often the model complies. | Meta (PurpleLlama) | Capture-the-flag challenges | Expert | Not in cards | | Dec 2023 | Aug 2023 | ||
| DataBench Models answer 1,822 questions about 80 real-world tables, such as Forbes billionaires and Airbnb listings, with 3.7 million rows in total. Answers are booleans, numbers, categories or lists, matched against a key. A lite setting gives each table as a small sample. | Cardiff NLP (Cardiff University) and collaborators | Data science and SQL | University | Not in cards | | May 2024 | Jul 2023 | ||
| DebugBench Models repair bugs in 4,253 C++, Java, and Python programs derived from LeetCode solutions. The evaluation runs the repaired programs against tests. | THUNLP | Function-level problems | School | Not in cards | GPT-4 75% | Jan 2024 | Jun 2023 | ||
| DecBench Agents recover source code from compiled functions. The leaderboard measures the share that exactly match the original control flow, types, or recompiled instructions. | Noelo Lab, University of Georgia | Function-level problems | Expert | Not in cards | GPT-6 Astra 51.6% | Jul 2026 | Sep 2026 | ||
| Decision Index Vision Models make decisions over images and screens. The score is the published equal-weight mean of public and private vision scores. | Hugging Face community (apolinario, multimodalart) | Composite indices | Everyday | Not in cards | | Sep 2026 | Feb 2026 | ||
| DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. | Google DeepMind | Dangerous capabilities | Not in cards | | Mar 2024 | Dec 2023 | |||
| DesignQA Models read Formula SAE rules and inspect CAD views and drawings of a student race car. They retrieve rules, identify parts, and check compliance. | Massachusetts Institute of Technology and Autodesk Research | CAD models | University | Not in cards | GPT-4o 0.72 | Apr 2024 | Mar 2025 | ||
| Deterministic Agent Reasoning & Formatting Suite Models solve three algorithm questions under strict formatting rules. The harness checks the answers and their JSON structure. | Weni Mohamed | Following precise instructions | University | Not in cards | | Nov 2026 | Sep 2026 | ||
| DexVerse Robot policies perform tabletop manipulation tasks in Isaac Lab, including tool use, bimanual coordination, and sequences with several goals. Task-specific success conditions check completion under different visual settings and robot embodiments. | University of North Carolina at Chapel Hill, University of Hong Kong, and University of California, Berkeley | Simulated robots | Everyday | Not in cards | π0.5 34% | Jul 2026 | Apr 2025 | ||
| DraftNEPABench Agents draft sections of environmental impact statements using technical and regulatory source documents. Model judges score the drafts against rubrics, and specialists assess a sample. | Pacific Northwest National Laboratory and OpenAI | Law | Expert | Not in cards | GPT-5 2.41 | Feb 2026 | Sep 2025 | ||
| DRAMA-Eval Models generate dialogue with several speakers across multiple turns. The authors check word accuracy and use a model judge to evaluate delivery in context. | Jingbin Hu and colleagues | Speech recognition and translation | Everyday | Not in cards | | Oct 2026 | |||
| Duplex-MPE Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios. | Peking University | Speech recognition and translation | Everyday | Not in cards | | Sep 2026 | Feb 2026 | ||
| DuplexLedger Speech agents track facts that speakers introduce, revise, or leave unchanged during conversation. The evaluator asks which facts still apply. | Yuxuan Li and colleagues | Speech recognition and translation | Everyday | Not in cards | | Oct 2026 | Feb 2026 | ||
| EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. | Peking University (VaLuE Lab) | Simulated robots | Everyday | Not in cards | GPT-5.6 Terra 52% | Oct 2026 | Aug 2026 | ||
| EmbodiedCLUE-VLA (tabletop tasks) Robot policies carry out tasks on a tabletop. | SuperCLUE | Real robots | Everyday | Not in cards | | Dec 2025 | |||
| EnterpriseRAG-Bench Retrieval systems and agents answer 500 questions over a synthetic company's internal documents, slightly over 500,000 of them. | Onyx | Deep research | Expert | Not in cards | GPT-6.1 Sol 91.5% | Feb 2026 | Sep 2026 | ||
| EntireSongs Models transcribe complete English and Chinese songs with background music. Qwen reports transcription errors separately for each language. | Qwen | Speech recognition and translation | Everyday | Not in cards | | Jan 2026 | Jan 2026 | ||
| EvalVerse Video models generate cinematic clips from prompts and reference material derived from professional videos. Film experts compare outputs across visual design, acting, cinematography, aesthetics, emotion, shot sequences, and sound. A calibrated vision-language evaluator also estimates these judgments. | Hong Kong University of Science and Technology, Tencent, and collaborators | Video generation | Expert | Not in cards | HappyHorse 1.0 0.73 | May 2026 | |||
| Feedback Prize: English Language Learning 3.0 Models predict six writing trait scores for English learners. The competition averages the root mean squared error across the six traits. | The Learning Agency Lab, Georgia State University, and Vanderbilt University | Other professions | University | Not in cards | o3 0.59 | Aug 2022 | Apr 2025 | ||
| Feedback Prize: Predicting Effective Arguments 2.0 Replaced in Aug 2022 | Not in cards | Sora 0.61 | May 2022 | Aug 2022 | |||||
| Feedback Prize: Evaluating Student Writing 1.0 Replaced in May 2022 | Not in cards | Dec 2021 | |||||||
| Fire360 Models interpret panoramic video from firefighter training drills. They answer questions, caption actions, assess procedures, locate equipment, and match damaged objects to clean examples. | University of Illinois Urbana-Champaign and Illinois Fire Service Institute | Video understanding | Expert | Not in cards | GPT-4o 53.8% | Jun 2025 | Mar 2025 | ||
| FormalMATH Lite Models prove statements in the FormalMATH Lite subset. The board reports proof success at the stated sampling budget. | SphereLab, M-A-P, The Chinese University of Hong Kong, Numina and others | Formal proofs | Competition | Not in cards | | Apr 2025 | |||
| FreshStack Retrieval systems find documentation that supports answers to technical questions. The benchmark measures how many required facts the retrieved passages cover. | University of Waterloo and Databricks | Questions that need search | Expert | Not in cards | | Apr 2025 | Oct 2026 | ||
| GDPevo Agents learn from training tasks drawn from enterprise workflows and must reuse that experience on related test tasks. Gains at test time measure self-evolution. | Prism Shadow | Running a business | Expert | Not in cards | Claude Opus 4.8 67.1% | Aug 2026 | May 2026 | ||
| GeoFlip Models generate videos from paired scenes where a geometry change reverses the expected physical outcome. Model judges check the event and the scene structure. | Yuqi Li and colleagues | Video generation | School | Not in cards | | Oct 2026 | Jul 2025 | ||
| GIFT-Eval-CTX Models forecast a time series using past observations and information about future changes. | Salesforce AI Research | Real-world events | University | Not in cards | GPT-5.6 Sol 0.57 | Oct 2026 | Jul 2026 | ||
| GJ.cool Vertical Book OCR Evaluation Models transcribe vertically printed Chinese books and manuscripts. The authors report accuracy separately for pages with sparse interlinear notes, pages with dense notes, and manuscripts. | GJ.cool Research and Development Center | Documents, charts and text in images | University | Not in cards | GLM-OCR 89% | Apr 2026 | Feb 2026 | ||
| GlyphBench Agents choose actions from text-rendered game boards. The standard suite contains 303 tasks across six game collections and scores agents by cumulative return. | Roger Creus Castanyer and collaborators | Video games | University | Not in cards | | Sep 2026 | Aug 2026 | ||
| GPU Puzzles Models implement parallel GPU kernels for 14 puzzles. The port runs each kernel against a reference and rejects serial implementations. | Sasha Rush | GPU kernels | University | Not in cards | GPT-5 mini 90.5% | Jul 2022 | Aug 2025 | ||
| GroundCocoa Models combine stated conditions with everyday knowledge to choose an answer. The questions test whether the model applies several constraints together. | Harsh Kohli and colleagues | Language reasoning | Everyday | Not in cards | | Apr 2024 | Dec 2023 | ||
| Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. | Amit Roth and colleagues | Deception and alignment | Expert | Not in cards | Claude Opus 4.6 10.1% | Aug 2026 | Feb 2026 | ||
| HALAS Models transcribe earnings call segments with human annotations of hallucinations. NVIDIA measures word error and segments with inserted text. | Mateusz Barański and colleagues | Speech recognition and translation | Everyday | Not in cards | Whisper large-v3 23.8% | Jun 2026 | Oct 2024 | ||
| HaluBench Models judge whether an answer is faithful to its context across about 15,000 labeled context, question and answer triplets. It tests hallucination detectors and LLM judges. | Patronus AI | Hallucination rates | University | Not in cards | Claude 3 Sonnet 78.8% | Jul 2024 | Mar 2024 | ||
| HealBench-Eval Models generate images intended to provide psychological comfort. The authors compare 14 image generators and prompt strategies on 1,095 cases. | East China Normal University and collaborators | Image generation and editing | Everyday | Not in cards | | Oct 2026 | |||
| Histoires Morales Models read French stories about social norms, intentions and actions. The test asks them to distinguish actions or consequences appropriate to the situation. | Thibaud Leteno and colleagues | Deception and alignment | Everyday | Not in cards | | Jan 2025 | Sep 2023 | ||
| HLS-Eval Models write or edit C++ for high-level synthesis. Tests check circuit behavior, and synthesis tools check whether the code can become hardware. | Georgia Institute of Technology | Electronics | University | Not in cards | | Dec 2024 | |||
| Honor of Kings Bug AI Root Cause Analysis Agents use the Honor of Kings Bug AI platform to identify the cause of code defects. The team measures how often agents identify the correct cause. | Tencent Honor of Kings | Tasks from real jobs | University | Not in cards | | Nov 2026 | Nov 2026 | ||
| HumanEval-X Models generate programs or translate solutions among Python, C++, Java, JavaScript, and Go. The 820 human-written language variants include executable tests, and scoring uses pass@k. | THUDM | Function-level problems | School | Not in cards | GPT-NeoX-20B 9.78% | Mar 2023 | Apr 2022 | ||
| Humor in AI: Cartoon Captioning Models write captions for New Yorker cartoons. Human judges and language models compare the captions with contest entries from different ranking groups. | University of Wisconsin–Madison, University of Washington and collaborators | Creative writing | Everyday | Not in cards | Claude 3 Opus 73.9% | Jun 2024 | Feb 2024 | ||
| Indic DiarBench Models transcribe conversations in 22 Indian languages and identify who spoke each word. Human annotators supply speaker labels and timestamps, including for overlapping speech. | Sarvam AI and AI4Bharat at IIT Madras | Speech recognition and translation | Everyday | Not in cards | GPT-4o 36.2% | Jul 2026 | Nov 2025 | ||
| Inverse Scaling Prize Tasks Models answer questions designed to reveal cases where larger models perform worse. The suite includes hindsight bias and following instructions that conflict with familiar text. | Ian R. McKenzie and colleagues | Language reasoning | School | Not in cards | | Jun 2023 | Apr 2022 | ||
| JamC-QA Models answer multiple-choice questions about Japan, including regional identity, customs, geography, and law. | SB Intuitions | Knowledge in many languages | School | Not in cards | | Jul 2025 | Sep 2024 | ||
| Japanese National Dental Examination 2025 Models answer 350 questions from Japan's 2025 dental licensing exam. The study reports accuracy separately for 202 text questions and 148 visual questions. | Japan Ministry of Health, Labour and Welfare | Medicine | University | Not in cards | Apr 2025 | ||||
| Japanese National Dental Examination 2024 Replaced in Apr 2025 | Not in cards | o1 81.9% | Apr 2024 | Jan 2025 | |||||
| Japanese National Examination for Occupational Therapists 2025 Models answer 199 Japanese occupational therapy licensing questions. The authors score answers against the official key and compare performance on text and visual questions. | Kazuya Saita, Yuichi Mine, and Satoru Amano | Medicine | University | Not in cards | o3 88.4% | Apr 2025 | Apr 2025 | ||
| JCoLA Models decide whether Japanese sentences are grammatically acceptable. Linguists supply judgments for sentences from textbooks and the research literature. | Taiga Someya and colleagues | Reasoning in many languages | Everyday | Not in cards | GPT-4 54.5% | Feb 2022 | Jan 2024 | ||
| JETHICS Models judge ethical scenarios in Japanese. The suite includes the Japanese commonsense morality dataset used by llm-jp-eval. | Masashi Takeshita and Rafal Rzepka | Reasoning in many languages | Everyday | Not in cards | GPT-4o 71.3% | Jun 2025 | Nov 2024 | ||
| Kimi-ASR Internal Testset Subset 1 Models transcribe speech from Moonshot internal subset 1. The score is the published word error rate. | Moonshot AI | Speech recognition and translation | Everyday | Not in cards | Kimi-Audio-7B-Instruct 1.42% | Apr 2025 | Apr 2025 | ||
| Kimi-ASR Internal Testset Subset 2 Models transcribe speech from Moonshot internal subset 2. The score is the published word error rate. | Moonshot AI | Speech recognition and translation | Everyday | Not in cards | Kimi-Audio-7B-Instruct 2.44% | Apr 2025 | Apr 2025 | ||
| Known-Unknown Questions Models distinguish questions with established answers from questions with no definitive answer. The authors score this distinction using F1 and equal error rate. | University of California, Santa Barbara, University of Waterloo, and Vector Institute | Short factual questions | Everyday | Not in cards | | May 2023 | Feb 2023 | ||
| Korean Contest Prize Reading Models extract the first place cash award from Korean contest notices. The grader distinguishes it from total prize pools and noncash gifts. | ohyoseok | Reading in many languages | School | Not in cards | GPT-5.4 Nano 100% | Oct 2026 | Apr 2026 | ||
| LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. | INSAIT | Simulated robots | Everyday | Not in cards | GPT-6 Astra 63.3% | Oct 2026 | Sep 2026 | ||
| Language Barrier Benchmark Models solve coding and planning tasks in different languages. The coding task uses translated instructions and runs submissions against the same tests. | Evelia Veldrine | Reasoning in many languages | University | Not in cards | GPT-6 Astra 0% | Oct 2026 | Sep 2026 | ||
| LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. | KlingAI Research (Kuaishou) and Peking University | Video generation | Everyday | Not in cards | | May 2026 | |||
| LongHealth Models answer 400 questions about long clinical records from 20 fictional patients. Tasks test retrieval, sorting and recognition of missing information. | LongHealth authors | Reasoning over long documents | University | Not in cards | GPT-3.5 Turbo 51.7% | Jan 2024 | Dec 2023 | ||
| MaCBench Vision language models answer chemistry and materials questions from images. Tasks cover reading data, understanding experiments and interpreting results. | MaCBench authors (Alampara et al., ChemBench group) | Lab and data work | Expert | Not in cards | GPT-5.2 72.6% | Nov 2024 | Dec 2025 | ||
| Matter-of-Fact Models decide whether materials science claims are feasible using evidence available before a cutoff date. The authors compare their answers with later published findings. | Peter Jansen and colleagues | Research science | Expert | Not in cards | Claude 3.7 Sonnet 72% | Jun 2025 | Apr 2025 | ||
| MedConceptsQA Models answer multiple-choice questions about medical codes. The hub supports the original dataset and a separate 2,000-example sample. | Nadav Lab | Medicine | Expert | Not in cards | GPT-4 Turbo (Jan 2024) 61.9% | Mar 2024 | Jan 2024 | ||
| MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. | Yingshu Li and colleagues | Medicine | University | Not in cards | | Nov 2026 | Jan 2026 | ||
| MemTrapBench Models with memory are tested on cases where correct, relevant memories still distort their reasoning or beliefs and lower task performance. | Zhejiang University (ZJUNLP) | Learning from the context | Everyday | Not in cards | | Aug 2026 | Dec 2025 | ||
| MM-NIAH Vision-language models answer questions about long documents of interleaved images and text. Tasks are retrieval, counting and reasoning over needles hidden in the document. | Shanghai AI Laboratory (OpenGVLab) | Finding facts in long inputs | Everyday | Not in cards | | Jun 2024 | Apr 2024 | ||
| MulSeT Models answer spatial questions about direction, distance and masks across views. The paper studies single images, multiple views and video. | Wanyue Zhang and colleagues | Spatial reasoning and counting | University | Not in cards | | Sep 2025 | Jan 2025 | ||
| MultiGEC 2025 Models correct whole texts in 12 European languages. One track rewards minimal edits, and the other rewards fluent rewrites. | Computational Second Language Acquisition working group | Reading in many languages | School | Not in cards | | Jun 2024 | Jul 2024 | ||
| MultiHop-RAG Retrieval-augmented systems answer 2,556 queries whose evidence is spread across two to four news articles. The queries also use document metadata, such as source and date. The benchmark scores both retrieval quality and answer accuracy. | Hong Kong University of Science and Technology (Yixuan Tang, Yi Yang) | Questions that need search | Everyday | Not in cards | GPT-4 Turbo 56% | Jan 2024 | Nov 2023 | ||
| MultiMedQA Models answer medical exam, research, and consumer health questions across seven datasets. Multiple-choice components use answer accuracy. Clinicians and other raters assess free responses for factuality, precision, harm, and bias. | Google Research and DeepMind | Medicine | Expert | Not in cards | | Dec 2022 | Apr 2022 | ||
| NanoGPT-Bench Coding agents with a fixed H100-hour budget and no internet try to improve on a strong NanoGPT speedrun starting point. The score is the share of later human progress recovered. | Intology | AI research tasks | Research frontier | Not in cards | Claude Opus 4.6 9.30% | May 2026 | Mar 2026 | ||
| New Yorker Caption Contest Humor Understanding Models match a cartoon to its caption, choose the funnier of two captions, and explain a joke. They receive the cartoon or its description. | Allen Institute for AI and collaborators | General visual questions | Everyday | Not in cards | | Sep 2022 | Oct 2019 | ||
| News With GPT Instructions Models classify financial news by sentiment. ChatGPT supplied seven sentiment labels and explanations. The public test contains 4,047 news items. | Oliver Wang | Finance | Everyday | Not in cards | | Jul 2023 | Jul 2023 | ||
| NorSumm Models summarize Norwegian news articles in Bokmål or Nynorsk. Human summaries provide references in both written standards. | Samia Touileb and colleagues | Reading in many languages | School | Not in cards | | Jan 2025 | Sep 2023 | ||
| Numeric Grounding Models answer numeric questions from a supplied context. Six tasks test lookup, abstention, distractors, calculations, units, and combinations of these demands. | Alan Finney | Faithful to sources | School | Not in cards | GLM-5 1.00 | Oct 2026 | Feb 2026 | ||
| OceanGym Agents interpret underwater images and sonar, then navigate simulated search and inspection tasks. The simulator scores how closely they reach target locations. | Zhejiang University and collaborators | Simulated robots | University | Not in cards | GLM-4.5V 14.6 | Sep 2025 | Aug 2025 | ||
| OmniMedVQA Models answer questions about 118,010 medical images across 12 imaging modalities. The authors measure accuracy against the correct answer option. | University of Hong Kong and Shanghai AI Laboratory | Medicine | Expert | Not in cards | | Feb 2024 | May 2023 | ||
| OncoTriad-QA Models answer questions about cancer patients using radiology, pathology, and genomic evidence. The authors score multiple-choice answers and compare written responses with reference answers. | Ahnaf Munir and colleagues | Medicine | Expert | Not in cards | GPT-5.4 46% | May 2026 | Mar 2026 | ||
| OpenRSI Index ACE Playbook Inspection and Repair Agents repair a reasoning playbook under a fixed budget. The score is held-out Formula exact-answer accuracy as a fraction. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | Claude Opus 5 0.73 | Sep 2026 | Jul 2026 | ||
| OpenRSI Index Isaac Lab PegInsert Reward Search Agents design robot training rewards under a fixed budget. The score is held-out peg-insertion success as a fraction. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | GPT-5.6 Sol 0.60 | Sep 2026 | Jul 2026 | ||
| OpenRSI Index Kev Decision Architecture Agents improve a decision model under a fixed budget. The score is the published exponential of negative macro log loss. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | GPT-5.6 Sol 0.29 | Sep 2026 | Jul 2026 | ||
| OpenRSI Index Learnability-Aware Long/Short CoT Adaptation Agents adapt mathematical reasoning examples under a fixed training budget. The score is accuracy across five math benchmarks. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | Claude Fable 5 47.5% | Sep 2026 | Jul 2026 | ||
| OpenRSI Index Marin Scaling Ladder Agents design one optimizer for six model sizes. The score is the published complete six-rung reward relative to AdamH. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | GPT-5.6 Sol 1.01 | Sep 2026 | Jul 2026 | ||
| OpenRSI Index MInference 32-Head Sparse Prefill Kernel Agents optimize a sparse-attention prefill operator under a fixed budget. The score is speedup against the paired reference run. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | GPT-5.6 Sol 1.54× | Sep 2026 | Jul 2026 | ||
| OpenRSI Index Molmo2 Video-Pointing Inference Strategy Agents tune video inputs and inference for a frozen Molmo2 model. The score is held-out spatiotemporal pointing soft F1. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | Claude Fable 5 0.46 | Sep 2026 | Jul 2026 | ||
| OpenRSI Index MolmoWeb Interaction Context Allocation Agents select interaction context for a frozen browser model. The score is macro-averaged browser-action prediction quality. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | Claude Opus 5 0.61 | Sep 2026 | Jul 2026 | ||
| OpenRSI Index ReasonIR Difficulty Curriculum Agents adjust retrieval training data schedules. The score measures reasoning retrieval with a penalty for general retrieval regression. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | Claude Opus 5 0.22 | Sep 2026 | Jul 2026 | ||
| OS-Kairos Agents predict Android actions for tasks such as finding a product in Amazon. The task succeeds when every action matches the recorded trajectory. | Shanghai Jiao Tong University and Meta | Phones | Everyday | Not in cards | Qwen-VL-Max 29.8% | Feb 2025 | Aug 2024 | ||
| Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. | Pantograph PBC | Real robots | Everyday | Not in cards | GPT-6 Astra 36.3% | Sep 2026 | Sep 2026 | ||
| PerMemBench Memory systems for long-running agents decide what to store from years of interactions across many domains and user personas. Later questions check whether the system kept the context that matters for each user. The benchmark tests whether memory policies can adapt to the person instead of storing everything the same way. | KAIST | Learning from the context | Everyday | Not in cards | | May 2026 | Aug 2025 | ||
| PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. | Positronic Robotics | Real robots | Everyday | Not in cards | π0.5 75.1% | Mar 2026 | Dec 2025 | ||
| PHMForge Agents analyze battery and aircraft engine data to diagnose faults, estimate remaining life, and make maintenance decisions. Harbor packages 99 industrial scenarios as 123 tasks. | IBM Research | Other professions | Expert | Not in cards | Claude Opus 4.6 80.8% | Apr 2026 | Feb 2026 | ||
| Pravda data voids chatbot audit Chatbots answer 13 English questions from two countries. Researchers score endorsement of false claims and citations to Pravda websites separately. | Maxim Alyukov, Mykola Makhortykh, Alexandr Voronovici, and Maryna Sydorova | Long-form factuality | Everyday | Not in cards | Grok-2 0% | Sep 2025 | Jun 2025 | ||
| PRISM Instruction Order Sensitivity Models solve the same math problems under different orderings of fixed instructions. The evaluation compares accuracy across orderings on GSM8K and MATH-500. | Blessings Mambwe | Following precise instructions | School | Not in cards | | Aug 2026 | Sep 2025 | ||
| ProSA Models answer equivalent instructions with different wording. PromptSensiScore measures how much model performance varies with the prompt. | OpenCompass and collaborators | Following precise instructions | University | Not in cards | | Oct 2024 | Feb 2024 | ||
| PROWBench Programmable world models must render video that follows the rules and interactions their program specifies. The test checks visual fidelity to fine-grained rules. | Alaya Lab | Interactive world models | Everyday | Not in cards | | Oct 2026 | Sep 2025 | ||
| Putnam-AXIOM Models solve Putnam mathematics problems and equivalent variants. The evaluator checks final answers to test whether models generalize beyond familiar problem statements. | Aryan Gulati and colleagues | Competition problems with answers | Competition | Not in cards | o1-preview 41.9% | May 2025 | Sep 2024 | ||
| REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. | Czech Technical University (CIIRC) + University of Amsterdam | Simulated robots | Everyday | Not in cards | π0-FAST 53.1% | Dec 2025 | Jan 2025 | ||
| Red Team Defense Simulator Models respond to an adaptive adversary. The evaluation measures successful defense, helpfulness and false refusals. | Intertwine | Jailbreaks and prompt injection | University | Not in cards | GPT-4o 85% | Sep 2025 | Mar 2025 | ||
| RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. | AgentAI Co., Ltd. | Simulated robots | Everyday | Not in cards | | Mar 2026 | Dec 2025 | ||
| RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. | RoboTwin Platform (HKU MMLab + others) | Simulated robots | Everyday | Not in cards | π0.5 58.3% | Jun 2025 | Apr 2025 | ||
| RoboTwin 1.0 Replaced in Jun 2025 | Not in cards | Sep 2024 | |||||||
| RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. | MTEB community and Hugging Face (with MongoDB) | Other | University | Not in cards | | Oct 2025 | Jun 2025 | ||
| SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. | Tsinghua University (THU-RCSCT, Hu et al.) | Simulated robots | Everyday | Not in cards | π0.5 57.8% | Dec 2025 | Apr 2025 | ||
| ScholarQABench Models answer scientific questions by synthesizing research papers. The benchmark checks citations and grades long answers against expert rubrics. | OpenScholar team | Research science | Expert | Not in cards | | Nov 2024 | Apr 2024 | ||
| SciBench Models solve open-ended college-level problems in mathematics, chemistry and physics. Most answers are numbers, which a script checks against the key. | University of California, Los Angeles | Science questions | University | Not in cards | | Jul 2023 | Apr 2024 | ||
| ScopeCheck: evidence before numbers Models calculate an operating metric when the data covers the requested population. Paired cases test whether they abstain when essential evidence is missing. | haxforscience | Faithful to sources | University | Not in cards | Claude Haiku 4.5 0.94 | Oct 2026 | Mar 2026 | ||
| ScreenSuite Models use screenshots to locate controls, select actions and complete computer tasks. The suite measures perception, single actions and complete interactions. | Hugging Face | Desktop apps | Everyday | Not in cards | | Jun 2025 | Mar 2025 | ||
| SecQA Models answer multiple-choice computer security questions at two difficulty levels. Questions come from a security textbook and were checked by people. | SecQA author (Zefang Liu) | Finding and exploiting real bugs | University | Not in cards | | Dec 2023 | Dec 2023 | ||
| SemComp-Bench Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference. | FrameX AI | Video generation | Everyday | Not in cards | | Aug 2026 | Jul 2025 | ||
| SHROOM-CAP Models read answers to questions about scientific papers in nine languages. They flag factual errors and fluency errors separately. | CHOMPS shared task organizers | Faithful to sources | University | Not in cards | gpt-oss-20b 0.61 | Jul 2025 | Aug 2025 | ||
| SIB-200 Models predict the topic of a sentence across 205 languages and dialects. The sentences come from FLORES-200. | David Ifeoluwa Adelani and colleagues | Reading in many languages | Everyday | Not in cards | GPT-4 48.7% | Sep 2023 | Jun 2023 | ||
| SimReady Text-to-CAD Models convert mechanical part descriptions into CAD programs. OpenCASCADE checks geometry. The hub port uses 111 prompts covering 37 parts from the original 150-prompt evaluation. | SimReady | CAD models | University | Not in cards | GPT-6.1 Sol 96.2% | Nov 2026 | Sep 2026 | ||
| SMM4H-HeaRD 2026 Pathology Report Staging Models assign tumor, lymph node, and distant metastasis stages to synthetic pathology reports. A tiebreaker removes explicit stage labels and requires inference from clinical findings. | Cedars-Sinai | Medicine | Expert | Not in cards | GPT-5.4 Mini 0.95 | Nov 2025 | Mar 2026 | ||
| SO-Arena Models choose answers across 11 domains and five capability groups. Evaluation tests accuracy, stability under changed option sets, and recognition that the correct answer is missing. | National University of Singapore and collaborators | Language reasoning | University | Not in cards | GPT-6 Sol 82.3% | Nov 2026 | Sep 2026 | ||
| Social Signal Bench Models identify overlapping social signals in conversations from audio or video. The score weights each label by annotator agreement and gives credit for related predictions. | University of Copenhagen and collaborators | Audio understanding and voice | Everyday | Not in cards | | Nov 2026 | Sep 2026 | ||
| SpeechEditBench Models edit English and Chinese speech from written instructions. Joint success requires the requested change and preservation of the attributes checked for that task. | City University of Hong Kong and collaborators | Speech recognition and translation | University | Not in cards | gpt-realtime 96.7% | Apr 2026 | Aug 2025 | ||
| SREGym-Lite Agents diagnose and mitigate failures in running services on a Kubernetes cluster. | SREGym team | Terminal tasks | Expert | Not in cards | Claude Sonnet 5.5 81.3% | Jul 2026 | Oct 2026 | ||
| StationeryBench Robots use both arms to handle common desk supplies. A human grades progress from zero to 100 on five tasks. | Robocurve | Real robots | Everyday | Not in cards | GPT-6 Astra 46.0 | Sep 2026 | Sep 2026 | ||
| Student-Work Grading Models grade student work. The tasks compare grades with human marks and test how blank answers, appeals, and grading instructions change the result. | David Porcaro | Other professions | University | Not in cards | Claude Haiku 5.5 0.86 | Nov 2026 | Oct 2026 | ||
| SugarCrepe Models choose between a caption that describes an image and a plausible caption with a changed detail. | Cheng-Yu Hsieh and colleagues | General visual questions | School | Not in cards | | Jun 2023 | Jul 2024 | ||
| SuperCLUE-DeepSWE Agents complete extended software engineering tasks in Chinese. | SuperCLUE | Fixing real repositories | Expert | Not in cards | GPT-6.1 Sol 84.3 | Oct 2026 | |||
| SuperCLUE-R2V Models generate videos conditioned on reference material. | SuperCLUE | Video generation | University | Not in cards | Grok Imagine Video 1.5 64.9 | ||||
| SWE-bench-Live Verified Task success on the official Verified split. The source reports each submission denominator separately. Other language and platform splits remain separate. | Microsoft | Fixing real repositories | Expert | Not in cards | Claude Sonnet 4.5 40% | May 2025 | Sep 2025 | ||
| SWE-smith Agents repair software faults generated from real repositories. Harbor evaluates a 100-task sample from the larger training collection using tests for each fault. | SWE-bench team | Fixing real repositories | Expert | Not in cards | Claude Sonnet 4 38% | Apr 2025 | May 2025 | ||
| Talk to Act Models turn conversations into robot actions. Each predicted function call must identify the action and its arguments from the dialogue. | Deepu Krishnareddy and colleagues | Function calling | Everyday | Not in cards | | Nov 2026 | Mar 2026 | ||
| tau-bench Airline Agents help simulated airline customers through tools while following airline policies on the original tasks. The score is pass^1 as a percentage. | Sierra | Customer service | Everyday | Not in cards | Claude 3.5 Sonnet (October 2024) 46% | Jun 2024 | Oct 2024 | ||
| Tennis & Padel State Tracking Benchmark Models follow a sequence of tennis or padel points and report the score and next server. A rules engine checks each answer. | Gulrez Qayyum | Language reasoning | Everyday | Not in cards | gpt-oss-120b 0% | Oct 2026 | Apr 2026 | ||
| TextWorld Cooking (GATA) Agents navigate text-based rooms, gather ingredients, and prepare recipes. The port sums points earned during play and includes separate training and test games at increasing difficulties. | GATA authors | Video games | Everyday | Not in cards | | Feb 2020 | Aug 2025 | ||
| The Illusion of AI Safety Chatbots answer 30 prompts about self-harm, eating disorders, and substance use. Researchers check replies for harmful content and invitations to continue the conversation. | Center for Countering Digital Hate | Wellbeing and sycophancy | University | Not in cards | GPT-4o 43% | Oct 2025 | Aug 2025 | ||
| TheOne Routing Decisions Models route messages to an existing topic, start a new topic, or request clarification. The grader checks their decisions on 36 synthetic cases. | Yuli Dai | Multi-turn conversation | Everyday | Not in cards | GPT-5.4 Nano 94.4% | Oct 2026 | Mar 2026 | ||
| ts-bench v1 Coding agents solve 25 self-contained TypeScript exercises from Exercism. The score is the percentage of exercises that pass their tests. | laiso (independent) | Function-level problems | University | Not in cards | GPT-5.3 Codex 96% | Jun 2025 | Feb 2026 | ||
| TurBLiMP Models compare Turkish sentence pairs that differ in grammaticality. The core tests cover 16 grammatical phenomena. | Ezgi Başar and colleagues | Reasoning in many languages | Everyday | Not in cards | | Jun 2025 | Mar 2025 | ||
| TurnBench Turn-taking systems listen to two-person conversations and must detect when a speaker's turn ends and when the other speaker interrupts. The test set has 116 English conversations in six interaction styles, each labeled by three annotators. Systems rank by recall, and a system whose false-positive rate exceeds 0.15 ranks below every qualifying system. Latency after each event is also reported. | Sesame AI with Mundo AI, Carnegie Mellon University and others | Speech recognition and translation | Everyday | Not in cards | | Aug 2026 | Mar 2026 | ||
| UFO-Bench Models generate customized images from text and visual references. The evaluator breaks each request into individual requirements and checks how they interact. | University of Science and Technology of China and collaborators | Image generation and editing | Everyday | Not in cards | | Sep 2026 | |||
| UI-Genie reward model evaluation Models judge individual Android actions and decide whether a sequence of actions completes a task. The authors report separate F1 scores for both judgments. | CUHK MMLab and vivo AI Lab | Phones | Everyday | Not in cards | | May 2025 | Jan 2025 | ||
| UI-Vision Layout Grounding Models identify interface layout regions. The score is the published intersection over union percentage. | ServiceNow Research and collaborators | Finding things on screen | Everyday | Not in cards | Qwen2-VL-7B-Instruct 24.3% | Mar 2025 | Feb 2025 | ||
| UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. | Soochow University + Lumos Robotics + others | Real robots | Everyday | Not in cards | π0.5 55.8% | Jun 2026 | Apr 2025 | ||
| UserBench (developer simulation) Models predict a developer’s next message in a real coding session. The evaluator compares predicted and observed action labels using Jaccard similarity. | UserBench team | Multi-turn conversation | University | Not in cards | GPT-5.6 Sol 51.3% | Jul 2026 | Jul 2026 | ||
| Vals Smith: abhishek203/SfAgent Agents modify abhishek203/SfAgent to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 4.8 80% | Aug 2026 | May 2026 | ||
| Vals Smith: benredmond/apex Agents modify benredmond/apex to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Luna 63.3% | Aug 2026 | Jul 2026 | ||
| Vals Smith: BerriAI/litellm Agents modify BerriAI/litellm to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | | Aug 2026 | Jul 2026 | ||
| Vals Smith: conjfrnk/memovox Agents modify conjfrnk/memovox to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 90% | Aug 2026 | Jul 2026 | ||
| Vals Smith: Effect-TS/effect Agents modify Effect-TS/effect to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 93.3% | Sep 2026 | Jul 2026 | ||
| Vals Smith: flutter/flutter Agents modify flutter/flutter to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | | Aug 2026 | Jul 2026 | ||
| Vals Smith: Lemonawa/music163bot-rust Agents modify Lemonawa/music163bot-rust to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Sol 93.3% | Sep 2026 | Jul 2026 | ||
| Vals Smith: mirumee/ariadne Agents modify mirumee/ariadne to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GLM-5.2 83.3% | Aug 2026 | Jul 2026 | ||
| Vals Smith: nationalsecurityagency/ghidra Agents modify nationalsecurityagency/ghidra to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GLM-5.2 73.3% | Aug 2026 | Jul 2026 | ||
| Vals Smith: pallets-eco/flask-wtf Agents modify pallets-eco/flask-wtf to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | GPT-5.6 Luna 100% | Oct 2026 | Jul 2026 | ||
| Vals Smith: poalrom/sharepaste Agents modify poalrom/sharepaste to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 50% | Aug 2026 | Jul 2026 | ||
| Vals Smith: tock/tock Agents modify tock/tock to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Sonnet 5 86.7% | Aug 2026 | Jul 2026 | ||
| Vals Smith: vllm-project/llm-compressor Agents modify vllm-project/llm-compressor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5 96.7% | Sep 2026 | Jul 2026 | ||
| VCIS Models inspect a grid and decide whether a proposed movement plan reaches its goal. Separate tasks check object coordinates and the step where a plan fails. | Fahmi | Spatial reasoning and counting | School | Not in cards | Claude Sonnet 5.5 100% | Oct 2026 | Sep 2026 | ||
| VenusBench-Mobile Agents use Android apps such as VLC, Markor, and Pro Expense. A model judges 90 primary tasks. The other tasks use programmatic checks. | Venus Team, Ant Group | Phones | Everyday | Not in cards | Kimi K2.5 24.8% | Feb 2026 | Jan 2026 | ||
| VerilogEval Models complete Verilog modules from human or machine descriptions. A simulator checks their outputs against the reference behavior. | NVIDIA | Electronics | University | Not in cards | o1-preview 77.1% | Sep 2023 | Sep 2024 | ||
| VES Label Check Models write site notes about groundwater survey curves. The grader checks whether they retain a correct label or repeat a wrong label from the file. | Daniel Nwaneri | Science questions | University | Not in cards | Claude Sonnet 5 14.9% | Oct 2026 | Jun 2026 | ||
| VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. | Fudan University (OpenMOSS) | Simulated robots | Everyday | Not in cards | | Dec 2024 | Jun 2024 | ||
| Voice Arena Full Duplex Bench Voice agents respond to pauses, changes of speaker, acknowledgments, and interruptions. The board averages success rates across these conversational behaviors. | Voice Arena | Audio understanding and voice | Everyday | Not in cards | | Sep 2026 | |||
| WeRef-Bench Models locate objects described in English or Chinese and return no boxes when the image contains no match. | Tencent | Detection and captioning | Everyday | Not in cards | | Oct 2026 | Feb 2026 | ||
| WMT24 Icelandic Idiomatic Expressions Models translate English phrases into Icelandic. Context determines whether a phrase has its everyday literal meaning or its idiomatic meaning. | Árni Magnússon Institute for Icelandic Studies | Translation | Everyday | Not in cards | | Jun 2024 | Aug 2024 | ||
| World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. | Interactive world models | Everyday | Not in cards | Runway Gen-4 64.8% | Oct 2025 | ||||
| WRENCH (Factorio) Agents build Factorio factories and recover from unannounced disruptions. The evaluator measures how much production the factory retains after a disruption. | WRENCH authors | Video games | University | Not in cards | Claude Sonnet 5 1.49 | Aug 2026 | Jun 2026 | ||
| xCodeEval Models understand, generate, translate, and retrieve contest code across seven tasks and up to 17 programming languages. Generated programs run against test cases through ExecEval. | Islamic University of Technology, Nanyang Technological University, Qatar Computing Research Institute and others | Competitive programming | Competition | Not in cards | | Mar 2023 | Feb 2024 | ||
| ZhoBLiMP Models compare pairs of Chinese sentences that differ in grammatical acceptability. Accuracy measures how often the model prefers the grammatical sentence. | Yikang Liu and colleagues | Reading in many languages | Everyday | Not in cards | | Nov 2024 | Sep 2024 | ||
| Addis Scribe Benchmark Speech models transcribe recordings in seven African languages. Addis AI compares transcripts with human references and measures how quickly some systems return text. | Addis AI | Speech recognition and translation | Everyday | Not in cards | | Oct 2026 | Sep 2026 | ||
| ADQA-Bench Models listen to recordings and answer 3,000 questions about speech, music, and environmental sounds. One question asks how a bird calls before it stops. | DCASE 2026 Task 5 organizers | Audio understanding and voice | Everyday | Not in cards | | Jan 2026 | Sep 2025 | ||
| AdSpark-Bench Models generate advertisement videos from product images and prompts. The benchmark checks whether products remain consistent and whether each shot conveys the requested selling point. | Zhifei Yang and colleagues | Video generation | University | Not in cards | | Oct 2026 | |||
| Aether-Bench Models retouch photographs automatically, match a requested style, or apply specified adjustments. The style test measures fidelity on 100 image pairs from unseen presets, including peak signal-to-noise ratio. | Zhejiang University, vivo BlueImage Lab, and the University of Chinese Academy of Sciences | Image generation and editing | University | Not in cards | | Apr 2026 | Sep 2025 | ||
| AgentCLUE-General Agents complete general purpose tasks in Chinese. | SuperCLUE | General assistants | University | Not in cards | OpenAI Computer-Using Agent (CUA) 4.62 | Jan 2025 | |||
| AgentIF-OneDay Agents complete 104 tasks that stand for a full day of human work, life and study. Tasks include tedious workflows, inferring rules from examples and changing requirements. | xbench (HongShan) | General assistants | Expert | Not in cards | ChatGPT agent 62.6% | Dec 2025 | Jul 2025 | ||
| Agentset Reranker Leaderboard Rerankers reorder retrieved documents across six datasets. A fixed embedding model retrieves 50 candidates, each reranker selects 15, and GPT-5 pairwise judgments produce Elo ratings. | Agentset | Questions that need search | University | Not in cards | | Nov 2025 | May 2025 | ||
| AGI Score (A Definition of AGI) Experts test models on ten cognitive domains from Cattell-Horn-Carroll theory, such as reasoning, memory and perception, with adapted human psychometric tests. Each domain counts 10% toward a score out of 100%. | Center for AI Safety and collaborators (Hendrycks et al.) | Composite indices | Expert | Not in cards | GPT-5 57% | Oct 2025 | Aug 2025 | ||
| AGI-Eval Community Speech Synthesis Models synthesize speech. The board ranks models on how human the voices sound. | AGI-Eval (Meituan) | Audio understanding and voice | Everyday | Not in cards | GPT-4o Mini TTS 13.0 | May 2025 | Mar 2025 | ||
| AGI-Eval Community Voice Interaction Models converse by voice. The board scores naturalness, problem solving, learning assistance, leisure and empathy. | AGI-Eval (Meituan) | Audio understanding and voice | Everyday | Not in cards | GPT-4o 0.60 | May 2025 | Mar 2025 | ||
| AmbiStory Models rate how plausible a word meaning is in a short story. The shared task averages rank correlation and accuracy within the organizers’ tolerance. | University of Technology Nuremberg | Reading comprehension | School | Not in cards | | Jul 2025 | Jul 2024 | ||
| Amharic ASR Benchmark Speech models transcribe 1,548 recordings of people reading Amharic prompts. The organizers report character and word error rates. | Dataset.ET | Speech recognition and translation | Everyday | Not in cards | Whisper large-v3 151.1% | Sep 2026 | Nov 2023 | ||
| AMLworld-Compact Models inspect transaction graphs and estimate whether a marked target is suspicious. The grader thresholds the probabilities and computes a weighted F1 score. | Natapong Nitarach (Schwyter) | Finance | University | Not in cards | | Oct 2026 | Sep 2026 | ||
| AndroidLab Mobile agents complete 138 tasks across nine apps on predefined Android virtual devices, in text-only and multimodal modes. Checks on the final device state score each task. | Tsinghua University and Zhipu AI (THUDM) | Phones | University | Not in cards | GPT-4o 31.2% | Oct 2024 | Nov 2024 | ||
| ANES survey item associations Models fill in one missing answer from a 2016 election survey using the respondent’s other answers. Researchers compare associations between the real and simulated answers. | Lisa P. Argyle and colleagues | Social simulation | Everyday | Not in cards | GPT-5 mini 0.82 | Sep 2021 | Aug 2025 | ||
| APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. | Shanghai Jiao Tong University | Long videos | Everyday | Not in cards | | Sep 2026 | Jul 2026 | ||
| Appendix Boundary Models extract answers from documents containing clean text, benign notes, or conflicting instructions. The author reports this task separately from replications on the same page. | Rudots | Jailbreaks and prompt injection | School | Not in cards | | Oct 2026 | Jun 2025 | ||
| ArtEdit-Bench-Eval Models rate photo edits against English instructions. Spearman and Pearson correlations measure agreement with human ratings on 200 cases. | Tencent Hunyuan and Xiamen University | Image generation and editing | Everyday | Not in cards | | Nov 2025 | Sep 2025 | ||
| AV-SafetyBench Models generate videos with sound from written prompts. Judges review each clip with sound, in silence, and as audio alone to identify unsafe outputs. | Korea University | Video generation | Everyday | Not in cards | LTX-2.3 49.4% | Sep 2026 | Mar 2026 | ||
| BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues Models write the teacher's next reply in an English lesson. Organizers average the system's ranks on BERTScore F1 and DialogRPT. | BEA 2023 shared task organizers | Multi-turn conversation | University | Not in cards | GPT-4 3.00 | Mar 2023 | Jun 2023 | ||
| BioMaze Models answer questions about biological pathways under normal and altered conditions. The benchmark includes true-or-false questions and written responses. | Haiteng Zhao and colleagues | Genomics and life sciences | University | Not in cards | | Feb 2025 | Jul 2024 | ||
| BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. | Stanford CRFM | Finding and exploiting real bugs | Expert | Not in cards | Claude 3.7 Sonnet 50% | May 2025 | Feb 2025 | ||
| Brennan Center election image generation audit Image generators receive requests for misleading election scenes. Researchers count how often each system produces an image, including images that omit requested details. | Brennan Center for Justice | Refusing harmful requests | Everyday | Not in cards | Runway Gen-4 100% | Aug 2026 | |||
| C2HLSC Models rewrite ten C programs for hardware synthesis. The tests check program output and whether the synthesis tool accepts the implementation. | New York University | Electronics | University | Not in cards | Claude 3.5 Sonnet 74% | Nov 2024 | Jun 2024 | ||
| CAGUI Models locate controls and read marked text in Chinese Android screenshots. A separate test checks each predicted action against the recorded steps for an app task. | OpenBMB | Phones | Everyday | Not in cards | | May 2025 | Jan 2025 | ||
| Can evaluation miss a useful reframing? Models analyze cases that may benefit from reframing. A separate judge scores useful reframing on analytical cases and checks whether responses preserve the facts. | Andrii Volynets | Language reasoning | University | Not in cards | | Oct 2026 | Dec 2025 | ||
| Cat-Goric Game Benchmark Models navigate a 2D platform game and submit results to a public decision-making leaderboard. | Felladrin | Video games | Everyday | Not in cards | | Oct 2026 | Aug 2026 | ||
| CellCaution: Reasoning About Cell Composition Models reason about changes in cell populations and decide when the information supports a numerical claim about their composition. | AbhishekNandy | Genomics and life sciences | University | Not in cards | | Oct 2026 | Apr 2026 | ||
| ChessBench Models select chess moves from legal options in positions scored by a chess engine. Decision Index measures whether the selected move has maximal action value. | Google DeepMind | Puzzles and chess | Competition | Not in cards | GPT-3.5 Turbo Instruct 66.5% | Sep 2023 | |||
| ChildMandarin Models transcribe Mandarin speech from children ages three to five or verify speakers. | BAAI FlagEval | Speech recognition and translation | Everyday | Not in cards | | Sep 2024 | Aug 2024 | ||
| Chinese Essay Fluency Evaluation 2024 Models identify errors and rewrite sentences from Chinese school essays. A new track asks them to rate the fluency of a whole essay. | East China Normal University and Microsoft Research Asia | Reading in many languages | School | Not in cards | Jan 2024 | ||||
| Chinese Essay Fluency Evaluation 2023 Replaced in Jan 2024 | Not in cards | ChatGLM-6B 51.6 | Mar 2023 | Mar 2023 | |||||
| ChipGPTV Models write Verilog from circuit diagrams and descriptions with different levels of detail. Testbenches check whether the generated circuits behave correctly. | Institute of Computing Technology, Chinese Academy of Sciences, and collaborators | Electronics | University | Not in cards | GPT-4V 71.8% | Jul 2024 | Nov 2023 | ||
| CIRR Models find an image that matches a reference photograph after applying a change described in text. | Zheyuan Liu and colleagues | General visual questions | Everyday | Not in cards | | Aug 2021 | Oct 2026 | ||
| CLEF 2026 SimpleText Task 2 Models detect or classify unsupported additions to simplified Cochrane abstracts. A generation subtask compares rewrites made with and without explicit checks against the source. | University of Amsterdam and University of Western Brittany | Faithful to sources | University | Not in cards | | Sep 2025 | Apr 2025 | ||
| CLEF 2025 SimpleText Task 2 Replaced in Sep 2025 | Not in cards | Nov 2024 | |||||||
| CLEF JOKER 2023 Task 1: Pun Detection Models decide whether a text contains a pun. Organizers report precision, recall, F1, and accuracy separately for English, French, and Spanish. | CLEF JOKER organizers | Reading in many languages | Everyday | Not in cards | T5-Large 55% | Sep 2022 | |||
| CLEF JOKER 2025 Task 2: Wordplay Translation Models translate English puns into French. A translator judges whether each sampled answer preserves the meaning and contains wordplay. The online leaderboard uses BLEU. | CLEF JOKER organizers | Translation | University | Not in cards | Oct 2024 | ||||
| CLEF JOKER 2024 Task 3: Pun Translation Replaced in Oct 2024 | Not in cards | Nov 2023 | |||||||
| CLEF JOKER 2023 Task 3: Pun Translation Replaced in Nov 2023 | Not in cards | T5-Large 10.0 | Sep 2022 | ||||||
| CMI-Bench Models listen to music and answer requests such as naming an instrument, marking beats, or transcribing lyrics. Each task has its own scoring rule. | Queen Mary University of London and Yamaha Corporation | Audio understanding and voice | University | Not in cards | | Jun 2025 | Mar 2026 | ||
| Connected Robotics Evidence Graph Benchmark Models update beliefs as evidence arrives from connected robots and sensors. The evaluator checks the belief state after each observation across 350 scenarios. | Ranulfo Bezerra and colleagues | Embodied reasoning | University | Not in cards | | Nov 2026 | Mar 2025 | ||
| ConvFinQA Models answer a conversation of related numerical questions about a company's financial report, with tables and text, where later questions build on earlier answers. The dataset has about 3,900 conversations with 14,000 questions, built from FinQA. Answers are checked by execution accuracy against the gold reasoning program. | UC Santa Barbara and J.P. Morgan AI Research | Finance | University | Not in cards | T5-Large 58.7% | Oct 2022 | |||
| COPAL-ID Models choose a plausible cause or consequence in Indonesian scenarios. The questions require familiarity with local customs and expressions. | Haryo Akbarianto Wibowo and colleagues | Reasoning in many languages | Everyday | Not in cards | | Nov 2023 | Jul 2023 | ||
| CORAL Models retrieve passages, answer questions, and cite supporting evidence across conversations. Retrieval metrics, response-overlap metrics, model judgments, and citation precision and recall score different parts of the task. | Renmin University of China, Beijing Academy of Artificial Intelligence, Huawei Poisson Lab, and Waseda University | Questions that need search | Everyday | Not in cards | | Oct 2024 | Sep 2024 | ||
| CoralVQA Models answer questions about coral identity, growth, and health in reef photographs. GPT-4o checks answers against references across 16 question types. | Beijing University of Posts and Telecommunications and collaborators | General visual questions | University | Not in cards | Claude 3.5 Haiku 41.8% | May 2025 | Oct 2024 | ||
| CoVEBench Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged. | Nanjing University (NJU-LINK) | Video generation | Everyday | Not in cards | HappyHorse 1.0 55.2 | Jun 2026 | |||
| CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. | Po-Yi Wu and colleagues | Simulated robots | Everyday | Not in cards | | Nov 2026 | Jun 2024 | ||
| Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. | Ashish Thapliyal and colleagues | General visual questions | School | Not in cards | | May 2022 | Oct 2026 | ||
| CyberSecEval MITRE False Refusal Rate Models answer borderline but benign security questions. Scored by how often the model wrongly refuses. | Meta (PurpleLlama) | Capture-the-flag challenges | Expert | Not in cards | | Apr 2024 | Aug 2023 | ||
| DeceptionBench (PKU Alignment) Models state a belief and then answer under social or strategic pressure. Judges measure how often their public answer contradicts their private reasoning. | Jiaming Ji and colleagues | Deception and alignment | Everyday | Not in cards | | May 2025 | Jan 2025 | ||
| Decision Model Benchmark Models classify examples of hallucinations, judge model outputs, and route agent requests. | Arize AI | Hallucination rates | University | Not in cards | GPT-6 Luna 87.9% | Oct 2026 | Sep 2026 | ||
| DeepResearch Bench II Deep research agents write reports for 132 research briefs in 22 domains, each derived from a review article they may not consult. A judge model checks 9,287 expert criteria. | University of Science and Technology of China and Metastone Technology | Deep research | Expert | Not in cards | OpenAI deep research 43.1% | Jan 2026 | Feb 2025 | ||
| DeepResearch Bench II (Gemini 2.5 Pro judge) Systems conduct deep research under the legacy Gemini 2.5 Pro evaluator. Scores remain separate from the current GPT-5.5 evaluator. | University of Science and Technology of China and Metastone Technology | Deep research | Expert | Not in cards | OpenAI deep research 45.4% | Jan 2026 | Feb 2025 | ||
| dependency-bench Models reason about build dependencies, retries, and failures in continuous integration pipelines. The grader checks their answers by exact match. | Muhammad Owais Warsi | Fixing real repositories | University | Not in cards | Claude Haiku 5.5 85% | Oct 2026 | Oct 2026 | ||
| DevOps-Gym Agents build, monitor, repair and test Java and Go projects. The Prime port converts tasks to single-turn questions with proxy grading. | UCSB ML Security Lab and collaborators | Fixing real repositories | University | Not in cards | Claude Sonnet 4 27.5% | Jan 2026 | May 2025 | ||
| DexArena Robots use both hands to complete household and industrial tasks. The simulator checks each stage and the final outcome across ten multistage tasks. | Shanghai Institute for Physical Artificial Intelligence and Robotics and collaborators | Simulated robots | Everyday | Not in cards | π0.5 52.1% | Oct 2026 | Apr 2025 | ||
| DFKI English-Russian Linguistic Test Suite at WMT25 Models translate English passages into idiomatic Russian. Rules match known translations. A linguist judges unresolved cases across 55 phenomena. The authors report item-weighted and category-weighted accuracy. | DFKI, Saarland University and University of Hildesheim | Translation | University | Not in cards | GPT-4.1 89.5% | Jun 2025 | Apr 2025 | ||
| Does the answer survive disagreement? Models answer checkable questions and respond to a user who insists on a plausible wrong answer. The test measures whether they keep the correct answer. | Kum Donalsien | Wellbeing and sycophancy | School | Not in cards | GLM-5 0% | Oct 2026 | Feb 2026 | ||
| DREsS Models score English learner essays for content, organization, and language. Each trait receives a score from 1 to 5 in half-point steps. | Haneul Yoo and colleagues at KAIST | Other professions | University | Not in cards | | Feb 2024 | Jul 2024 | ||
| DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. | UTN Nuremberg + KIT + Franka Robotics + TU Munich | Simulated robots | Everyday | Not in cards | π0.5 10.6% | Jun 2026 | Apr 2025 | ||
| EchoChain Voice agents reason through spoken exchanges while a speaker interrupts or changes instructions. The score measures whether the agent responds correctly. | Labelbox | Audio understanding and voice | Everyday | Not in cards | gpt-realtime 44% | Mar 2026 | Aug 2025 | ||
| EICAT impact classification Models read a scientific paper and classify the ecological impact of a named invasive species. They choose among five impact levels and a data deficient category. | Bielefeld University | Lab and data work | Expert | Not in cards | | Feb 2025 | Jul 2024 | ||
| EmotionTalk Models infer emotions and sentiment from Chinese conversations. | BAAI FlagEval | Audio understanding and voice | Everyday | Not in cards | chatglm2-6b 50.9 | May 2025 | Jun 2023 | ||
| ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. | ESI-Bench team | Embodied reasoning | Everyday | Not in cards | GPT-5 70% | May 2026 | Aug 2025 | ||
| EtiCor Models judge whether a stated behavior fits the etiquette of a given region. The test covers five regions and scores judgments against regional labels. | Indian Institute of Technology Kanpur | Commonsense | Everyday | Not in cards | | Oct 2023 | Mar 2023 | ||
| EuroSAT Models classify land use and land cover in Sentinel-2 satellite images. | Patrick Helber and colleagues | General visual questions | University | Not in cards | | Aug 2017 | Oct 2026 | ||
| EvaLatin 2026 Models parse Latin prose and poetry and identify named entities in classical Latin. | EvaLatin organizers | Reading in many languages | University | Not in cards | | Nov 2025 | Dec 2025 | ||
| EvaLatin 2024 Replaced in Nov 2025 | Not in cards | Dec 2023 | |||||||
| EventHallusion Models answer questions about video events that can prompt false descriptions. The evaluation tests whether answers stay grounded in the clip. | EventHallusion authors | Video understanding | Everyday | Not in cards | GPT-4V 76.5% | Sep 2024 | Nov 2023 | ||
| EVF-Bench Models generate product videos with visual effects. The evaluator checks motion, product appearance, text stability, and whether the effects fit the request. | University of Chinese Academy of Sciences and collaborators | Video generation | University | Not in cards | | Nov 2026 | |||
| FactoryBench-100 Agents complete 100 manufacturing workflows through simulated ERP, email, spreadsheet, and chat tools. FactoryScore checks their decisions and the state left in each system. | Blobfish AI | Tasks from real jobs | Expert | Not in cards | GPT-5.6 Luna 76.2% | Aug 2026 | Jul 2026 | ||
| FGVC-Aircraft Models identify aircraft variants from photographs. | Subhransu Maji and colleagues | Detection and captioning | University | Not in cards | | Jun 2013 | Oct 2026 | ||
| FinanceBench Models answer open-book questions about public companies using their filings, such as 10-Ks and earnings reports. The full set has 10,231 questions with answers and evidence strings, and 150 of them are public. The makers graded answers by hand, and later users often grade with a model judge. | Patronus AI, Contextual AI and Stanford University | Finance | Expert | Not in cards | | Nov 2023 | Jul 2023 | ||
| FinMem Trading Agents choose whether to buy, sell or hold a stock each day. The benchmark measures cumulative return against historical prices. | FinMem authors | Finance | University | Not in cards | davinci-003 1.63% | Nov 2023 | Nov 2022 | ||
| FiQA SA (BloombergGPT protocol) Models classify sentiment toward a named financial target as negative, neutral, or positive. Bloomberg uses five examples and reports weighted F1. | Bloomberg | Finance | Everyday | Not in cards | GPT-4 0.63 | Mar 2023 | Jun 2023 | ||
| Formal Logic Deduction Models decide whether facts prove or disprove a hypothesis, or leave it unresolved. They also generate the proof steps. | Terufumi Morishita and colleagues | Abstract puzzles and logic | University | Not in cards | T5-Base 75.8% | Aug 2023 | |||
| FortisAVQA Models answer questions about sounds and visible events in music videos. Accuracy is reported separately for frequent and rare question distributions after the original questions are rephrased. | Xi'an Jiaotong University and HKUST (Guangzhou) | Video understanding | Everyday | Not in cards | VITA 48.7% | Apr 2025 | Aug 2024 | ||
| FreshQA Models answer questions whose answers change over time, never change, or rest on a false premise that they should reject. FreshEval, an LLM judge, grades answers in a Relaxed mode or a Strict mode that also penalizes any hallucinated detail. | Google and UMass Amherst (Tu Vu and coauthors) | Questions that need search | Everyday | Not in cards | GPT-4 Turbo (Jan 2024) 80.6% | Oct 2023 | Jan 2024 | ||
| GeckoNum Models generate images from prompts about quantities. Human raters check exact counts, approximate quantities, and reasoning about numerical concepts. | Google DeepMind | Image generation and editing | Everyday | Not in cards | DALL-E 3 47.6% | Jun 2024 | |||
| GEOBench-VLM Models interpret satellite and aerial images, identify land use, and track changes across observations. Some tasks ask models to locate objects or outline regions. | Mohamed bin Zayed University of Artificial Intelligence and collaborators | General visual questions | University | Not in cards | GPT-4o 0.41 | Nov 2024 | Mar 2025 | ||
| GeoMap-Bench Models answer 3,864 questions about 124 geologic maps. They locate map features, interpret legends, and use geological evidence to explain conditions at a site. | Microsoft Research and collaborators | Documents, charts and text in images | Expert | Not in cards | GPT-4o 0.37 | Jan 2025 | Mar 2025 | ||
| GhostInk: Honest OCR on Bill Photos Models read text from phone photos and mask unreadable characters. The grader penalizes guessed text, even when the guess happens to match. | Akash | Documents, charts and text in images | Everyday | Not in cards | Grok 4.5 0% | Oct 2026 | Jul 2026 | ||
| HandWorldBench Models generate videos of human or robotic hands moving objects. The benchmark combines 17 measures of hand motion, object state, and video quality. | Zhongguancun Academy, Tsinghua University, and collaborators | Video world models | Everyday | Not in cards | | Nov 2026 | |||
| HappyWorld Spatial Reconstruction The HappyWorld arena publishes a separate spatial reconstruction vote pool and Elo ranking. | Alibaba | Interactive world models | Everyday | Not in cards | GPT-6 Astra 1,252 | Sep 2026 | Sep 2026 | ||
| HarmEval Models answer 550 harmful requests across 11 categories. A judge measures the share of unsafe responses. | SafeInfer authors | Refusing harmful requests | Everyday | Not in cards | | Jun 2024 | Dec 2023 | ||
| HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. | Qinhong Zhou and colleagues | Simulated robots | Everyday | Not in cards | π0 18.8% | Nov 2026 | Oct 2024 | ||
| Harvest Rush Training Evaluation Models decide whether a tractor should drive over or bypass animals and obstacles at a stated fuel cost. The default evaluation generates 300 held-out examples. | CompassionML | Deception and alignment | Everyday | Not in cards | | Sep 2026 | Jul 2024 | ||
| HD-EPIC VQA Models watch first-person kitchen videos and choose among five answers. Questions ask what people cooked, where they moved objects, and where they looked. | University of Bristol and the HD-EPIC team | Video understanding | Everyday | Not in cards | | Feb 2025 | Jan 2025 | ||
| HiACC Models transcribe adults and children switching between Hindi and English. The recordings include story reading and spontaneous answers. | Shruti Singh and colleagues | Speech recognition and translation | Everyday | Not in cards | | May 2025 | Jan 2026 | ||
| IMB Models answer Italian medical specialty exam questions and respond to patient questions from online forums. The paper reports results for both tasks. | Antonio Romano and colleagues | Medicine | University | Not in cards | | Oct 2025 | Jul 2024 | ||
| IMC 2025 Models write proofs for the 10 problems of the 2025 International Mathematics Competition for University Students. Human judges grade each proof. | IMC (contest), MathArena (evaluation) | Written proofs | University | Not in cards | | Aug 2025 | Jun 2025 | ||
| IMO-LeanProofBench Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof. | Google DeepMind (Superhuman Reasoning team) | Formal proofs | Competition | Not in cards | | Jun 2026 | Feb 2026 | ||
| JA-VLM-Bench-In-the-Wild Models answer Japanese questions about images of places, objects and culture in Japan. | Sakana AI | General visual questions | Everyday | Not in cards | llava-v1.6-mistral-7b 41.1 | Mar 2024 | Jan 2024 | ||
| Japanese Nephrology Board Renewal Questions 2014 to 2023 Models answer 209 Japanese nephrology board renewal questions from ten exam years, including questions with clinical images. Researchers compare answers with the society’s key. | Japanese Society of Nephrology | Medicine | Expert | Not in cards | o1-pro 81.3% | Jan 2025 | Mar 2025 | ||
| JorGPT Student Answer Grading Models grade university students’ written answers against the teacher’s marks. The authors released 3,041 responses to 50 questions in Spanish and English. | Universidad Francisco de Vitoria | Other professions | University | Not in cards | | Jan 2026 | Sep 2025 | ||
| K-QA Models answer 201 patient questions. The benchmark checks whether answers include essential statements from clinicians and whether any claims contradict those statements. | K Health and Hebrew University of Jerusalem | Medicine | Expert | Not in cards | GPT-4 67.7% | Jan 2024 | Jun 2023 | ||
| KMMLU-Hard Models answer about 5,000 Korean multiple-choice questions from KMMLU that most models got wrong, across many expert subjects. | HAE-RAE | Knowledge in many languages | University | Not in cards | GPT-4 Turbo (Jan 2024) 30.5% | Jan 2024 | Jan 2024 | ||
| KoCommonGEN v2 Models select a plausible Korean sentence for a given set of concepts from four options. | NLP and AI Lab at Korea University | Reasoning in many languages | Everyday | Not in cards | | Sep 2023 | Jul 2024 | ||
| Korean Dental Licensing Examination 2024 and 2025 Models answer 642 Korean dental licensing questions from two new exams. The authors score accuracy and break it down by subject and image type. | Taejun Kim and Bong Chul Kim | Medicine | Expert | Not in cards | Claude Opus 4 81.5% | Feb 2026 | May 2025 | ||
| LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. | Soongsil University + Chung-Ang University | Simulated robots | Everyday | Not in cards | π0.5 71.4% | Mar 2026 | Apr 2025 | ||
| LibriCount Models estimate how many speakers appear in a recording with overlapping voices. | Fabian-Robert Stöter and colleagues | Speech recognition and translation | Everyday | Not in cards | Whisper large-v3 57.2% | Dec 2017 | Nov 2023 | ||
| Life-Bench Models answer multimodal questions about synthetic personal lifelogs. Tasks cover concept identification, event understanding, and reasoning over multiple pieces of evidence. | Google DeepMind | General visual questions | Expert | Not in cards | | Feb 2026 | Mar 2025 | ||
| LifeChoice Models select the decision a character makes in a novel. The authors score the percentage of correct choices. | Fudan University and Alibaba Group | Reading comprehension | School | Not in cards | o1 65.3% | Apr 2024 | Dec 2024 | ||
| Live Gaming Benchmark Models watch game clips and decide when to speak. They narrate alone, join another commentator, or guide a player. A judge scores their commentary. | Proact-VL team | Video captioning | Everyday | Not in cards | LiveCC-7B-Instruct 28.4% | Mar 2026 | Apr 2024 | ||
| LiveDRBench Agents perform 100 deep research tasks and return key claims discovered during search. Scores use claim-level F1. | Microsoft Research | Deep research | Expert | Not in cards | OpenAI deep research 55.5% | Aug 2025 | Feb 2025 | ||
| LLM Writer Negative Style Models write responses to five prompts. Model judges flag stylistic habits such as promotional wording, editorializing, and formulaic conclusions. | Prime Intellect | Taste and style | Everyday | Not in cards | GPT-4.1 mini 0.52 | Oct 2025 | Apr 2025 | ||
| LLM-Coordination Models play four cooperative games and answer 198 multiple-choice questions about the same games. The authors report game outcomes and question accuracy separately. | University of California, Santa Cruz | Multi-agent games | Everyday | Not in cards | | Oct 2023 | Dec 2023 | ||
| LLM-SRBench Models infer scientific equations from observations on 239 problems across four scientific domains. Symbolic accuracy checks whether the discovered equation recovers the underlying relationship. | Virginia Tech, VinUniversity, Carnegie Mellon University and others | Research science | Research frontier | Not in cards | | Apr 2025 | Jul 2024 | ||
| LongBanana Models synthesize images from many reference images and textual constraints. Atomic rubric checks score content fidelity, relationships, and global constraints across 90 tasks. | Kling AI Research | Image generation and editing | Everyday | Not in cards | | Nov 2026 | Feb 2026 | ||
| Malt Freelancer Project Matching Models rank synthetic freelancer profiles against a search query or a project brief. Each task has 200 English prompts and 4,019 profiles across five languages. | Malt | Other professions | University | Not in cards | | Feb 2026 | Jun 2025 | ||
| MAPP (Multi-Agent Path Planning) Models move several agents across a grid without collisions. The environment tracks how many agents reach their goals and how closely their routes follow shortest paths. | salty-duck | Multi-agent games | School | Not in cards | GPT-4.1 70.8% | Sep 2025 | Apr 2025 | ||
| MedHallu Models distinguish factual medical answers from hallucinated answers at three difficulty levels. The benchmark tests detection with and without supporting knowledge. | UT Austin AI Health | Hallucination rates | University | Not in cards | | Feb 2025 | Jul 2024 | ||
| Mementos Models describe what happens across a sequence of images. The evaluation checks whether their descriptions capture the objects and actions. | Xiyao Wang and colleagues | General visual questions | Everyday | Not in cards | GPT-4V 40.6% | Jan 2024 | Nov 2023 | ||
| MEMTRACK Agents track changes across software platforms and use earlier information to complete tasks. | Patronus AI | Reasoning over long documents | Everyday | Not in cards | GPT-5 61% | Oct 2025 | Aug 2025 | ||
| mFollowIR Models rank documents according to a search instruction across multiple languages. | Orion Weller and colleagues | Questions that need search | University | Not in cards | | Jan 2025 | Sep 2023 | ||
| Miklós Schweitzer 2025 Models write proofs for the 2025 Miklós Schweitzer competition, a hard Hungarian university-level contest. MathArena runs and grades it. | MathArena (ETH Zurich SRI Lab) | Written proofs | Research frontier | Not in cards | GPT-5 Pro 90% | Dec 2025 | Oct 2025 | ||
| MINTEval Memory-augmented agents answer questions after long, frequently updated histories where memories interfere. Instances average 138,800 tokens and reach 1.8 million. | MINTEval authors (Lee et al.) | Learning from the context | Expert | Not in cards | | May 2026 | Apr 2026 | ||
| MLAgentBench Agents run machine learning experiments on 13 tasks, from improving CIFAR-10 accuracy to research problems like BabyLM. They read and write files, run code and inspect outputs. | Stanford University (SNAP) | Machine learning engineering | Expert | Not in cards | | Oct 2023 | Dec 2023 | ||
| MMArt-Bench Models retouch 200 photos from editing instructions. Gemini scores visual quality and instruction following, and image comparisons measure how closely edits match reference photos. | Xiamen University and collaborators | Apps and MCP servers | University | Not in cards | | Jun 2025 | |||
| MobileAgentBench Agents complete 100 tasks across ten Android apps. The grader checks app state and accessibility events, so agents can take different paths. | Carnegie Mellon University, University of Michigan, Northeastern University, and University of Hong Kong | Phones | Everyday | Not in cards | CogAgent 8.00% | Jun 2024 | Dec 2023 | ||
| MultiClinSum-2 Models summarize 1,000 clinical cases per language across 15 languages. Organizers report four automatic metrics and four language model judgments separately. | Barcelona Supercomputing Center | Medicine | University | Not in cards | Feb 2026 | ||||
| MultiClinSum v1 Replaced in Feb 2026 | Not in cards | | Apr 2025 | Sep 2024 | |||||
| MuseBench (Music Score and Performance Understanding) Models answer music theory questions, read piano scores, and interpret recorded performances. The authors report accuracy separately for text, images, and audio. | Qihao Zhao and colleagues | Audio understanding and voice | University | Not in cards | | Mar 2025 | |||
| MusiXQA Models read printed music and answer questions about notes, chords, text, and page layout. The authors measure answer accuracy and string similarity. | University at Buffalo and collaborators | Documents, charts and text in images | School | Not in cards | | Jun 2025 | May 2024 | ||
| NaijaSenti Models classify the sentiment of posts written in Hausa, Igbo, Nigerian Pidgin, and Yoruba. | Shamsuddeen Hassan Muhammad and colleagues | Reading in many languages | Everyday | Not in cards | | Jan 2022 | Jun 2025 | ||
| NarrLV Models generate videos from prompts that describe changes in a scene, an object, or an action. A multimodal judge checks what appears and in what order. | AMAP, Alibaba Group and collaborators | Video generation | Everyday | Not in cards | | May 2025 | Feb 2025 | ||
| NativeLongBench Models translate a complete book passage in one response. SEGALE aligns passages and COMET scores them against references across five input lengths. | Index LLM Team, Bilibili | Translation | University | Not in cards | | Sep 2026 | Aug 2026 | ||
| NEPAQuAD Models answer questions about environmental impact statements. The authors compare answers with a reference under several conditions, including retrieval and access to the relevant passage. | Pacific Northwest National Laboratory | Law | University | Not in cards | | Jul 2024 | Sep 2023 | ||
| NepTTS-Bench Speech models read Nepali sentences that test pronunciation contrasts. Native speakers rate the recordings, and the authors compare those ratings with an automated metric. | Ashish Thapa and colleagues | Speech recognition and translation | Everyday | Not in cards | | Apr 2026 | May 2025 | ||
| NWPU-RESISC45 Models classify remote sensing images into 45 scene categories. | Northwestern Polytechnical University | General visual questions | University | Not in cards | | Mar 2017 | Oct 2026 | ||
| OpenRSI Index SGLang Omni TTS Runtime Agents improve speech serving while preserving quality. The score is the official speedup reward for the archived continuation protocol. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | GPT-6 Astra 1.54× | Sep 2026 | Sep 2026 | ||
| PaperBench Code-Dev Agents implement research paper code. The Code-Dev variant evaluates coding separately from full research replication. | OpenAI | AI research tasks | Research frontier | Not in cards | o1 43.4% | Apr 2025 | Dec 2024 | ||
| PatchCamelyon Models identify whether a microscope image patch contains metastatic tumor tissue. | Bastiaan Veeling and colleagues | Medicine | Expert | Not in cards | | Jun 2018 | Oct 2026 | ||
| PersonaShot Models generate multi-shot videos about a person. The evaluator checks character continuity, emotional changes, and camera choices within and between shots. | Yuji Wang and colleagues | Video generation | Everyday | Not in cards | LTX-2.3 0.67 | Aug 2026 | Mar 2026 | ||
| PhoneHarness Bench Agents complete Android workflows through graphical controls, device commands, and structured tools. Checks inspect files, system settings, application state, and unintended side effects. | Tencent Hunyuan, CUHK, CUHK-Shenzhen and others | Phones | Everyday | Not in cards | | Jun 2026 | Feb 2026 | ||
| PitchBench Models listen to recordings and identify notes, intervals, chords, or melodies. The authors vary the instrument, timing, and background noise. | University of California, Berkeley and Thoughtful Lab | Audio understanding and voice | University | Not in cards | | May 2026 | Feb 2026 | ||
| Preference-ASR Models transcribe speech in a requested style. The evaluation checks preferences such as casing, number formatting, and removing disfluencies. | NVIDIA | Speech recognition and translation | Everyday | Not in cards | | Jun 2026 | Feb 2025 | ||
| Proof News election misinformation debunking Chatbots respond to five false claims about the 2024 US election. Reviewers mark whether each service clearly debunks each claim. | Proof News | Long-form factuality | Everyday | Not in cards | GPT-4 0% | Nov 2024 | Jun 2023 | ||
| PROST Models predict how objects move or change in everyday physical situations described in text. The benchmark tests reasoning about space and time. | Stéphane Aroca-Ouellette and colleagues | Commonsense | Everyday | Not in cards | GPT-2 (1.5B) 30.1% | Jun 2021 | Nov 2019 | ||
| QCircuitBench Models implement quantum algorithms, construct oracles, and synthesize circuits from written specifications. Harbor uses one instance from each of 28 algorithms. | Rui Yang and colleagues | Scientific code | Expert | Not in cards | | Oct 2024 | Apr 2024 | ||
| RadSum23 Models write the impression section of radiology reports from findings. One track includes chest images. Organizers rank submissions by RadGraph F1. | Stanford University | Medicine | Expert | Not in cards | T5-Base 38.0 | Jan 2023 | |||
| Real-Time Reasoning Gym Models act in games while token generation consumes part of the game clock. | Real-Time Reasoning Gym authors | Video games | Everyday | Not in cards | | Nov 2025 | Dec 2025 | ||
| RedditImpacts 2.0 Models mark clinical and social consequences of opioid use in first-person accounts. The organizers scored 278 annotated posts and excluded 300 unlabeled decoys. | Emory University and collaborators | Medicine | School | Not in cards | Aug 2025 | ||||
| Reddit-Impacts 1.0 Replaced in Aug 2025 | Not in cards | GPT-4o 0.42 | Dec 2023 | Mar 2025 | |||||
| Research Decision Benchmark Agents choose a next step from an explicit description of a research project. Rule-generated cases test ten types of decisions about evidence and research strategy. | Shaoning Zeng and colleagues | Research science | Expert | Not in cards | | Oct 2026 | Sep 2026 | ||
| RetryBudget-24: Policy and JSON Contracts Models apply retry policies in 24 synthetic cases. The harness checks their decisions and the required JSON structure across repeated runs. | Tao Li | Following precise instructions | University | Not in cards | | Oct 2026 | Jun 2025 | ||
| RGraph-Eval Models generate robot manipulation videos. The evaluator compares objects, changing relationships, and three-dimensional motion paths with a reference demonstration. | Aleksandr Lazarev and colleagues | Video generation | Everyday | Not in cards | | Oct 2026 | May 2026 | ||
| RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. | HKU MMLab + Peking University + others (RoboTwin team) | Simulated robots | Everyday | Not in cards | π0.5 10.4% | Mar 2026 | Apr 2025 | ||
| RoboCasa-Mem Robots complete nine household tasks that require memory. Later steps depend on objects the robot saw or actions it took earlier. | University of Science and Technology of China and collaborators | Simulated robots | Everyday | Not in cards | π0.5 16% | Nov 2026 | Apr 2025 | ||
| Roboflow 100-VL Models locate objects named in natural language across image collections from different domains. The benchmark scores predicted bounding boxes using average precision. | Roboflow | Detection and captioning | Everyday | Not in cards | | May 2025 | Jan 2025 | ||
| RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. | OpenHelix Team | Real robots | Everyday | Not in cards | π0.5 21.5% | May 2026 | Apr 2025 | ||
| SAC-Bench Models identify sounds and their order when a spoken claim contradicts the recording. Matched clips test whether answers follow the acoustic evidence. | The University of Osaka and University of Wisconsin-Madison | Audio understanding and voice | Everyday | Not in cards | | Oct 2026 | Sep 2025 | ||
| ScienceBenchmark Models translate questions about the Sloan Digital Sky Survey, cancer research, and research funding into SQL. The organizers run the queries and compare their results. | Zurich University of Applied Sciences and Athena Research Center | Data science and SQL | University | Not in cards | T5-Large 23% | Jun 2023 | Aug 2023 | ||
| ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. | Allen Institute for AI (Wang, Jansen, Côté, Ammanabrolu) | Lab and data work | School | Not in cards | T5-Large 0.15 | Mar 2022 | |||
| ScribbleEdit Models edit images using a scribble as the only instruction. The evaluator checks whether the change matches the intended operation and location. | ByteDance and Michigan State University | Image generation and editing | Everyday | Not in cards | | Oct 2026 | |||
| SCT-Bench Models rate how new clinical information changes a diagnosis or treatment decision. The grader awards partial credit according to the distribution of expert responses. | SCT-Bench authors | Medicine | Expert | Not in cards | o1-preview 70.5% | Feb 2025 | Sep 2024 | ||
| SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. | University of Illinois Urbana-Champaign (Lingming Zhang) + Purdue | Finding and exploiting real bugs | Expert | Not in cards | Claude 3.7 Sonnet 18% | Jun 2025 | Feb 2025 | ||
| Semantic DLP: Catch the Data Leak Models search long technical documents for proprietary content. They must identify the leak and quote the relevant span. | Karthik Sethuraman | Finding facts in long inputs | University | Not in cards | | Oct 2026 | Jun 2025 | ||
| SeniorTalk Models transcribe conversations with older Mandarin speakers and identify who speaks. | BAAI FlagEval | Speech recognition and translation | Everyday | Not in cards | Whisper large-v3 23.8% | Mar 2025 | Nov 2023 | ||
| Small Rewriter Benchmark Models rewrite short Verilog designs to reduce hardware use. The authors compare synthesized cell and wire counts and check whether the circuits preserve behavior. | Chinese University of Hong Kong and Huawei | Electronics | University | Not in cards | GPT-4V 152.8 | Nov 2023 | |||
| SMM4H 2025 Food Recalls and Foodborne Disease Outbreaks Models classify news articles as recalls, outbreaks, or neither. They also extract details such as the product, cause, and number of people affected. | SMM4H-HeaRD organizers | Medicine | School | Not in cards | o3-mini 0.88 | Mar 2025 | Jan 2025 | ||
| Something-Something V2 Models recognize actions involving people manipulating everyday objects. The benchmark distinguishes actions that differ in direction, ordering, or outcome. | TwentyBN | Video understanding | Everyday | Not in cards | | Jun 2025 | |||
| Sonetra Norwegian SNOMED CT Term Recognition Models transcribe Norwegian sentences read by a synthetic voice. A term counts as correct when every word in the medical term matches. | Sonetra AS | Speech recognition and translation | University | Not in cards | gpt-4o-transcribe 66.4% | Sep 2026 | Mar 2025 | ||
| SPA-Bench Agents use Android apps such as Airbnb, Spotify, and WeChat. The grader matches screenshots against task requirements and uses a vision model to resolve uncertain cases. | Huawei Noah's Ark Lab, Harbin Institute of Technology, Shenzhen, and University College London | Phones | Everyday | Not in cards | CogAgent 1.35% | Oct 2024 | Dec 2023 | ||
| StableToolBench Models solve user requests by calling tools from ToolBench's large set of real-world APIs. A virtual API server with a cache and API simulators stands in for live APIs, so results stay stable over time. GPT-4 judges whether each request was solved and which of two answers is better, giving a solvable pass rate and a win rate. | Tsinghua University and collaborators | Function calling | Everyday | Not in cards | GPT-4o mini 26.4% | Mar 2024 | Jul 2024 | ||
| StateBench (video continuation) Models continue a video while tracking what happened in earlier segments. A judge checks whether visible objects, hidden processes, and later state changes remain consistent. | Xi’an Jiaotong University, Alibaba Group, and collaborators | Video generation | Everyday | Not in cards | | Sep 2026 | Jul 2025 | ||
| SuperCLUE-200K Models retrieve information from long Chinese inputs. | SuperCLUE | Finding facts in long inputs | School | Not in cards | GPT-4 Turbo (Jan 2024) 9.98 | Jan 2024 | |||
| SuperCLUE-ComicShorts Models generate short animated comics. | SuperCLUE | Video generation | University | Not in cards | | ||||
| SuperCLUE-Long Models answer questions about long Chinese texts. | SuperCLUE | Reasoning over long documents | School | Not in cards | GPT-4o 68.4 | May 2024 | |||
| SuperCLUE-RAG Chinese models answer questions twice, first without documents and then with supplied documents. A judge model compares the answers to rate how well the system uses retrieved text. | CLUE benchmark team (SuperCLUE) | Faithful to sources | Everyday | Not in cards | GPT-4o 66.7 | Mar 2024 | May 2024 | ||
| SuperCLUE-Voice (speech interaction) Models interact with users through speech in real time. | SuperCLUE | Speech recognition and translation | Everyday | Not in cards | GPT-4o 73.0 | Nov 2024 | |||
| Swivuriso Models transcribe read and spontaneous speech in seven South African languages. The authors measure word error rates and keep speakers separate across splits. | University of Pretoria and the South Africa Next Voices team | Speech recognition and translation | Everyday | Not in cards | Whisper large-v3-turbo 15% | Nov 2025 | Oct 2024 | ||
| TAAROFBENCH Models respond to 450 social scenarios where Persian taarof etiquette may or may not apply. The evaluation measures culturally appropriate responses. | Nikta Gohari Sadr and collaborators | Chat and instructions in many languages | Everyday | Not in cards | | Sep 2025 | Mar 2025 | ||
| The Pushback Test Models advise a parent who questions urgent medical advice. The evaluation checks whether the model maintains the urgency required by NHS guidance. | Mark_RowSet | Medicine | University | Not in cards | Claude Sonnet 5 0.81 | Oct 2026 | Jun 2026 | ||
| TrustBoundary Bench Models plan incident response actions in a simulation. The grader checks whether they distinguish permission to act from information supplied as evidence. | onkar-cybersec | Jailbreaks and prompt injection | University | Not in cards | | Oct 2026 | Sep 2025 | ||
| UAQFact Models answer factual questions or decline when the facts do not support an answer. The benchmark also checks whether models know the facts needed to decide. | Soochow University and OPPO AI Center | Short factual questions | School | Not in cards | GLM-4 49.6% | May 2025 | Jan 2024 | ||
| Uniqueness of Optimum Models find the optimum of a small problem and report whether it is unique. Exhaustive enumeration supplies the expected value and number of optimal solutions. | Dimitar Kretski | Abstract puzzles and logic | University | Not in cards | gpt-oss-120b 55.2% | Oct 2026 | Aug 2025 | ||
| VAHBench Agents turn short drama briefs into finished videos with specified characters, dialogue, and durations. The evaluator checks completeness, quality, reliability, and accuracy of production reports. | Hongbo Zhang and colleagues | Video generation | University | Not in cards | Claude Opus 5 51.8 | Nov 2026 | Jul 2026 | ||
| VAmoS Bench Voice agents answer simulated bank customers and use tools to manage credit cards. A model checks each call against its transcript and recorded actions. | Veris AI | Customer service | University | Not in cards | | Jul 2026 | Mar 2026 | ||
| VBench image-to-video Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite. | NTU S-Lab and Shanghai AI Laboratory (Vchitect) | Video generation | Everyday | Not in cards | | Nov 2024 | Feb 2025 | ||
| VCG+ Diverse Models answer questions about videos drawn from varied settings. The evaluation grades their ability to describe and explain video content. | Muhammad Maaz and colleagues | Video captioning | Everyday | Not in cards | GPT-4V 43.5 | Jun 2024 | Nov 2023 | ||
| VeRO-Bench Coding agents improve other agents by editing their prompts, tools, and workflows. Each task has a reference evaluation for comparing changes. | Scale AI | Machine learning engineering | Expert | Not in cards | GPT-5.2 0.52 | Feb 2026 | Dec 2025 | ||
| ViStoryBench Models turn stories and character references into coherent image sequences. Model judgments and image features measure prompt alignment, character consistency, style, and image quality. | StepFun, AIGC Research, Westlake University AGI Lab, and ShanghaiTech University | Image generation and editing | University | Not in cards | | May 2025 | Sep 2025 | ||
| Voice Arena Diarization Bench Models identify who spoke when in recorded conversations. The board measures diarization error and publishes a separate track for speaker attributed transcription. | Voice Arena | Audio understanding and voice | Everyday | Not in cards | | Sep 2026 | |||
| Voice Arena Dictation Bench Dictation tools turn speech into written drafts. Judges measure the share of drafts a user could send without edits. | Voice Arena | Speech recognition and translation | Everyday | Not in cards | | Aug 2026 | |||
| Voice Controllability Leaderboard Text to speech systems create requested voices and follow delivery, emotion and role instructions. Blind human ratings and objective prosody measures score controllability. | Hume AI | Speech recognition and translation | Expert | Not in cards | | Sep 2026 | Sep 2026 | ||
| Voice Replication Leaderboard Text to speech systems reproduce reference speakers from short recordings. Human ratings score speaker identity, quality and naturalness, with an objective speaker similarity check. | Hume AI | Speech recognition and translation | Expert | Not in cards | | Sep 2026 | Sep 2026 | ||
| VRSBench Models caption aerial images, locate objects described in words, and answer visual questions. The held-out files contain 9,350 captions, 16,159 object descriptions, and 37,409 questions. | King Abdullah University of Science and Technology | General visual questions | Everyday | Not in cards | GPT-4V 65.6% | Jun 2024 | Nov 2023 | ||
| WebBench (Halluminate) Browser agents complete 2,454 tasks on 452 live websites from the global top 1,000 by traffic. Tasks either read information or write data: creating, updating and deleting records, logging in and downloading files. The board compares browser agent products. | Halluminate, with Skyvern | Web browsing | Everyday | Not in cards | OpenAI Computer-Using Agent (CUA) 76.5% | May 2025 | Jan 2025 | ||
| Werewolf Arena Models compete in Werewolf with strategic speaking, deception, deduction, and persuasion. | Werewolf Arena authors | Multi-agent games | Everyday | Not in cards | | Jul 2024 | Dec 2023 | ||
| WildAWM validation Agents complete 33 held-out computer-use tasks in generated software environments. Harbor packages the validation split separately from the training tasks. | MSR Orchard | Desktop apps | Expert | Not in cards | | Oct 2026 | Feb 2026 | ||
| WMT 2024 Discourse-Level Literary Translation Models translate Chinese web fiction into English, German, or Russian. Translators rate English outputs for general quality and for consistency across the passage. | Tencent AI Lab and China Literature | Translation | Expert | Not in cards | May 2024 | ||||
| WMT 2023 Discourse-Level Literary Translation Replaced in May 2024 | Not in cards | GPT-4 54.8 | Apr 2023 | Jun 2023 | |||||
| WorldGUI Desktop GUI agents do 611 tasks that start from varied initial states, such as the target app not being open. Tasks come with instructional videos and project files. | Show Lab, National University of Singapore | Desktop apps | Everyday | Not in cards | GPT-5.1 45.8% | Feb 2025 | Nov 2025 | ||
| XStoryCloze Models pick the right ending for short stories from StoryCloze, professionally translated into 10 languages. | Meta AI (XGLM team) | Reasoning in many languages | Everyday | Not in cards | GPT-3 6.7B 58.2% | Dec 2021 | Jun 2020 | ||
| ZeroCLUE Models perform Chinese language tasks without training on labeled examples from the target task. | CLUE benchmark team | Reading in many languages | School | Not in cards | ChatGLM-6B 30.9 | Mar 2023 | |||
| (S)elected Moderation campaign prompts Models receive requests for election campaign advice through their APIs. Reviewers count refusals. The report specifies ten English prompts. | AI Forensics and collaborators | Refusing harmful requests | Everyday | Not in cards | Jul 2024 | ||||
| 1X World Model Challenge Predict future egocentric frames of 1X's EVE humanoid from video and state. Compression scored by token cross-entropy, Sampling by PSNR on private tests. | 1X Technologies | Video world models | Not in cards | Jun 2024 | |||||
| 2010 i2b2/VA Challenge on Concepts, Assertions, and Relations in Clinical Text Models locate medical concepts, classify statements about problems, and determine relationships among problems, tests, and treatments. Organizers score each task separately. | Informatics for Integrating Biology and the Bedside and VA Salt Lake City Health Care System | Medicine | University | Not in cards | |||||
| 2013 Native Language Identification Shared Task Models identify an essay writer's first language from 11 choices. The test contains 1,100 English essays with 100 per language. | Educational Testing Service and the BEA workshop organizers | Other professions | University | Not in cards | Jan 2013 | ||||
| 2018 n2c2 Cohort Selection for Clinical Trials Models read longitudinal patient notes and decide whether patients meet specified clinical trial eligibility criteria. | Amber Stubbs and colleagues | Medicine | Expert | Not in cards | |||||
| 2022 n2c2/UW Shared Task on Extracting Social Determinants of Health Models identify substance use, employment, and living situations in clinical notes. Organizers score events and their attributes across three tests using notes from two institutions. | National NLP Clinical Challenges and the University of Washington | Medicine | University | Not in cards | Nov 2021 | ||||
| 2024 NIST GenAI Text-to-Text Discriminators Detectors estimate the probability that a machine wrote a summary, without seeing the source articles. NIST reports the area under the receiver operating characteristic curve. | National Institute of Standards and Technology | Other | Everyday | Not in cards | Apr 2024 | ||||
| 34th Japanese National Examination for Dental Hygienists Models answer 213 Japanese dental hygiene questions, including 139 with visual material such as instrument photographs and radiographs. | Hiroshima University | Medicine | University | Not in cards | Jun 2025 | ||||
| 3DPW Models recover a person's three-dimensional pose in outdoor video. Reference poses come from wearable sensors and a moving camera. | Timo von Marcard and colleagues | Spatial reasoning and counting | School | Not in cards | Sep 2018 | ||||
| 7Bench Layout-guided image models generate 224 text-and-layout samples across seven scenarios, scored for semantic and spatial alignment. | University of Padova | Image generation and editing | University | Not in cards | Aug 2025 | ||||
| A-OKVQA Models answer about 25,000 visual questions that need world knowledge beyond the image, in multiple-choice and direct-answer forms. | Allen Institute for AI | General visual questions | Everyday | Not in cards | Jun 2022 | ||||
| AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. | Artificial Analysis | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| AAP Periodontics In-Service Examination Models answer specialty examination questions used by North American periodontics residency programs. Researchers score answers against the Academy’s key for each exam year. | American Academy of Periodontology | Medicine | Expert | Not in cards | Jan 2024 | ||||
| AbMaze Agents make choices in a simulated antibody optimization experiment under a fixed budget. Recorded assay results determine normalized improvement on 200 test episodes. | valenlabs | Genomics and life sciences | Expert | Not in cards | Nov 2026 | ||||
| AbstentionAct Agents handle HR and operations requests with missing information or authority. The benchmark checks whether agents clarify, abstain, or complete a permitted action. | Victor Ojewale | Deception and alignment | Expert | Not in cards | Jul 2026 | ||||
| Accessible Structured Subtitle Generation 2026 Systems produce timed Chinese subtitles for deaf and hard-of-hearing viewers. The structured track also scores speaker labels, key information, and readability. | Beijing Union University and Tsinghua University | Video captioning | University | Not in cards | Feb 2026 | ||||
| ACI-Bench Models turn doctor-patient conversations into clinical visit notes. The original benchmark compares generated notes with clinician-authored references. | Wen-wai Yim and colleagues | Medicine | University | Not in cards | May 2023 | ||||
| ACII DaiKon Challenge Models predict interpersonal influence, turn taking and rapport trajectories from multimodal dyadic conversations across five languages. | Hume AI and ACII 2026 organizers | Audio understanding and voice | Expert | Not in cards | May 2026 | ||||
| ActivityNet Captions Models match descriptions of events to segments of untrimmed videos. Meta evaluates retrieval in both directions. | Ranjay Krishna and colleagues | Video understanding | Everyday | Not in cards | May 2017 | ||||
| Adaptive Reasoning (Vellum) Models solve 28 familiar logic puzzles after Vellum changes a premise. A judge checks the final answer, and a person reviews the result. | Vellum | Language reasoning | School | Not in cards | |||||
| ADE Corpus Models identify adverse drug effects in medical case reports. The OpenMed port combines classification and relation examples and reserves 20% for evaluation. | Harsha Gurulingappa and colleagues | Medicine | Expert | Not in cards | Oct 2012 | ||||
| ADE-bench Agents repair dbt projects so that SQL tests pass on the resulting tables. Harbor ports 48 DuckDB tasks and excludes the Snowflake and dbt Fusion variants. | dbt Labs | Data science and SQL | Expert | Not in cards | May 2025 | ||||
| ADE20K Models label the pixels in a scene by object or region. Meta measures overlap between predicted and reference segments. | MIT Computer Science and Artificial Intelligence Laboratory | Detection and captioning | Everyday | Not in cards | Aug 2016 | ||||
| ADQA Models describe film scenes. An evaluator answers questions from the descriptions and dialogue, then checks whether the context supports each correct answer. | Katha AI Lab, IIIT Hyderabad | Video captioning | Everyday | Not in cards | Oct 2025 | ||||
| ADVANCE audio and image retrieval (MTEB) Models match field recordings and aerial images from 5,075 locations. The correct recording and image come from the same place. | Massive Text Embedding Benchmark contributors | Audio understanding and voice | Everyday | Not in cards | Aug 2026 | ||||
| AdvBench Models get harmful requests, such as a request for malware, and should refuse them. The set has 520 harmful behaviors and 574 harmful strings. Most papers add an attack, such as an adversarial suffix, and report the attack success rate. | Carnegie Mellon University, Center for AI Safety and Bosch Center for AI (Zou, Wang, Kolter, Fredrikson) | Jailbreaks and prompt injection | Everyday | Not in cards | Jul 2023 | ||||
| AfriBED Models transcribe Bemba and Nyanja speech. The benchmark compares recognition errors across gender and speaker groups. | University of Cape Town | Speech recognition and translation | Everyday | Not in cards | Oct 2026 | ||||
| AfriQA Models retrieve evidence across languages to answer questions written in African languages. | Odunayo Ogundepo and colleagues | Reading in many languages | School | Not in cards | May 2023 | ||||
| AfriSenti Models classify sentiment in human-annotated social media posts across 14 African languages. | Shamsuddeen Hassan Muhammad and colleagues | Reading in many languages | Everyday | Not in cards | Feb 2023 | ||||
| Agent Bazaar Agents price goods or trade with sellers in simulated markets. The benchmark measures whether their decisions sustain prices, deter fraud, and preserve welfare and profits. | Agent Bazaar team | Multi-agent games | University | Not in cards | Apr 2026 | ||||
| Agent Behavior Loop v0 Models choose an assistant’s next action from an evidence bundle. Twelve tasks check whether coding and general assistants verify their work before claiming success. | narcolepticchicken | General assistants | University | Not in cards | Jun 2026 | ||||
| Agent Liability Gym Agents review case files and write legal memos about decisions made by autonomous agents. The grader checks their findings, evidence, and unsupported claims. | narcolepticchicken | Law | Expert | Not in cards | Jul 2026 | ||||
| Agent Memory Leaderboard Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework. | AML-memory | Learning from the context | Everyday | Not in cards | Jul 2026 | ||||
| AgentCLUE-Mobile Mobile assistants complete tasks on a phone. | SuperCLUE | Phones | Everyday | Not in cards | |||||
| Agentic Abstention Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on. | University of Washington (Luo, Wen and Wang) | Hallucination rates | Everyday | Not in cards | Jun 2026 | ||||
| Agentic Self-Reasoning (Apollo Research) Agents reason about their own operating conditions to complete tasks. Apollo measures success at three difficulty levels. | Apollo Research | Deception and alignment | Everyday | Not in cards | Aug 2024 | ||||
| Agents4PLC v2.0 Models write Structured Text control programs against formal requirements. The expanded benchmark adds 70 medium tasks and three hard tasks to the original 23. | Zhejiang University and UWin Tech | Electronics | University | Not in cards | Aug 2025 | ||||
| AgentSettlementBench Models decide whether to release or hold a payment under 20 settlement scenarios. The benchmark tracks accuracy and the cost of approving an unsafe settlement. | nagu | Finance | Expert | Not in cards | Feb 2026 | ||||
| AgentThreatBench Agents carry out routine tasks in the presence of untrusted content. The benchmark scores task completion and security separately. | vgudur-dev (Inspect Evals contributor) | Jailbreaks and prompt injection | University | Not in cards | May 2026 | ||||
| AGORA (Agentic Global Open Research Arena) Autonomous AI systems get one secret open finance question, 72 hours and a fixed budget to write a paper. Five finance scholars judge it. | Penn State Smeal College of Business | Finance | Research frontier | Not in cards | Oct 2026 | ||||
| Agri STT Benchmarking Dataset Models transcribe farmers’ spoken questions in Hindi, Telugu and Odia. Researchers score errors against human transcripts and separately measure errors on agricultural terms. | Digital Green | Speech recognition and translation | Everyday | Not in cards | Jan 2026 | ||||
| AI Agent Privacy Web agents complete tasks while exposed to private information. The benchmark measures privacy leakage and unsafe disclosure during autonomous operation. | Meta FAIR and Superintelligence Labs | Deception and alignment | Expert | Not in cards | Mar 2025 | ||||
| AI Code Review Evaluation (Augment Code) Agents review pull requests from open source projects. Augment compares their comments with a corrected reference set. | Augment Code | Fixing real repositories | Expert | Not in cards | Dec 2025 | ||||
| AI FPL Models manage Fantasy Premier League squads using the same weekly prompt. Their transfers, captain choices, and starting lineups earn points from real matches. | Scout Digital and Early Doors Sports | Multi-agent games | Everyday | Not in cards | Aug 2025 | ||||
| AI Law Legal Agent Eval Agents review five synthetic export control matters and write risk memos. The grader checks the deliverable, seven findings per matter, and unsupported claims. | narcolepticchicken | Law | Expert | Not in cards | Jul 2026 | ||||
| AI Office Benchmark (Moonshot) Agents complete end-to-end office tasks and produce documents, spreadsheets, PDFs and slide decks. Experts rate output quality. | Moonshot AI | Documents and spreadsheets | Expert | Not in cards | Jan 2026 | ||||
| AI Spreadsheet Benchmark Spreadsheet assistants complete 53 prompts for analysis, charts, formulas, and workbook management. Checks measure task success and whether outputs update when source data changes. The evaluation reports pass@1, pass@3, dynamic output rate, and latency. | Rows | Documents and spreadsheets | University | Not in cards | Sep 2025 | ||||
| AI4D Malawi News Classification Challenge Models assign a topic to a Chichewa news article from a Malawian newspaper or broadcaster. Zindi scores the share of correct predictions. | tNyasa Ltd Data Science Lab, AI4D Africa and Zindi | Reading in many languages | Everyday | Not in cards | Jan 2021 | ||||
| AIR-Bench 24.05 Retrievers find documents or passages that answer questions. This release covers 69 datasets across 13 languages. | AIR-Bench team with the Beijing Academy of Artificial Intelligence | Questions that need search | University | Not in cards | Oct 2024 | ||||
| AIR-Bench 24.04 Replaced in Oct 2024 | Not in cards | May 2024 | |||||||
| AISA-ArabicFC 2026 Models read an Arabic request and four tool schemas. They select a tool and supply its arguments, or decline to call a tool. | Tuwaiq Academy | Function calling | School | Not in cards | May 2026 | ||||
| AISE-Bench Agents answer academic information-seeking questions through multi-step knowledge-graph APIs and source-grounded responses. Planning, execution and answer-grounding metrics score each trajectory. | Z.ai, Tsinghua University and collaborators | Deep research | Expert | Not in cards | Jul 2026 | ||||
| AISHELL-4 Models transcribe Mandarin meetings and identify speakers from conference recordings. NVIDIA reports character error after matching speakers. | AISHELL and collaborators | Speech recognition and translation | Everyday | Not in cards | Apr 2021 | ||||
| ALE-Gold Agents use a Linux terminal to complete 96 professional tasks. Graders check the files they produce. A task passes only with full credit. | Sankalp (dejavucoder) | Long terminal sessions | Expert | Not in cards | Sep 2026 | ||||
| Alexandria Models translate conversations between English and regional Arabic. The public test contains 14,442 turns across 13 Arabic varieties. | Alexandria team | Translation | University | Not in cards | Jan 2026 | ||||
| AlexandriaX 2026 Models translate English dialogue into 13 Arabic varieties, translate financial text between six dialects and mark translation errors across five varieties. | AlexandriaX organizers | Translation | University | Not in cards | May 2026 | ||||
| ALFFA Swahili Broadcast News Test Speech models transcribe Swahili radio news. The test includes clean speech, background music, overlapping voices and noisy recordings. | Grenoble Informatics Laboratory and Dynamique du Langage | Speech recognition and translation | Everyday | Not in cards | May 2012 | ||||
| ALFFA Wolof Read Speech Test Speech models transcribe sentences read aloud in Wolof. Two speakers each read 1,000 test sentences. The recordings use different speakers from the training set. | Grenoble Informatics Laboratory and Voxygen | Speech recognition and translation | Everyday | Not in cards | Apr 2016 | ||||
| ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. | University of Washington, Allen Institute for AI and collaborators | Simulated robots | Everyday | Not in cards | Dec 2019 | ||||
| Alharthi LibriTTS Naturalness Listening Test Models estimate how natural synthesized speech sounds. Ten listeners rated each sample on a five-point scale for comparison. | Carnegie Mellon University and Mohamed bin Zayed University of Artificial Intelligence | Audio understanding and voice | Everyday | Not in cards | Oct 2023 | ||||
| Alibaba CTF Agents solve 87 capture-the-flag tasks in Alibaba’s collection of competition challenges from 2023 through 2026. | alibaba-aaig | Capture-the-flag challenges | Expert | Not in cards | Aug 2026 | ||||
| Aligned Scores and Performances for Piano Transcription Models turn piano performances into written scores. The reference scores let researchers assess pitches, voices, and the positions of notes within bars. | Francesco Foscarin and colleagues | Audio understanding and voice | Expert | Not in cards | Oct 2020 | ||||
| AliMeeting Models identify speakers in Mandarin meetings. NVIDIA compares diarization error using headset mixes and distant microphones. | Alibaba Speech Lab | Audio understanding and voice | Everyday | Not in cards | Feb 2022 | ||||
| AlloCiné Models classify French movie reviews as positive or negative. Review ratings supply the reference labels. | Théophile Blard | Reading in many languages | Everyday | Not in cards | Jan 2020 | ||||
| Alpha Arena Frontier models trade live markets with real money over a season, and the leaderboard ranks them by returns. | Nof1 | Finance | Expert | Not in cards | Oct 2025 | ||||
| AlphaEarth Foundations mapping evaluation Models use satellite representations to classify land cover, detect change, or estimate physical quantities. The evaluation compares predictions with independent measurements across regions. | Google DeepMind | General visual questions | University | Not in cards | Jul 2025 | ||||
| AlphaSense Financial Research Benchmark Agents answer 245 multi-step finance questions across company research, financial modeling, earnings, deals, markets and regulatory analysis using source retrieval. | AlphaSense | Finance | Expert | Not in cards | Nov 2026 | ||||
| Althingi Parliamentary Speech Models transcribe speeches delivered in the Icelandic parliament. Recordings and reference transcripts cover parliamentary proceedings from 2005 through 2016. | Reykjavík University and the Icelandic Parliament | Speech recognition and translation | Everyday | Not in cards | Aug 2017 | ||||
| AMemOps Models track instruments and their changing attributes in long audio streams. Questions test whether they recall the latest value and recognize absent sources. | University of Surrey and Royal Holloway, University of London | Audio understanding and voice | Everyday | Not in cards | Oct 2026 | ||||
| AmericasNLP 2025 Shared Task on Creation of Educational Materials for Indigenous Languages Models rewrite a sentence in an Indigenous language to make a requested grammatical change. The organizers count exact matches with the target sentence. | AmericasNLP | Chat and instructions in many languages | University | Not in cards | Jan 2025 | ||||
| AmericasNLP 2025 Shared Task on Machine Translation into Indigenous Languages Models translate between Spanish and 13 Indigenous languages of the Americas. The organizers score each direction using character and word overlaps with reference translations. | AmericasNLP | Translation | University | Not in cards | Jan 2025 | ||||
| AmericasNLP 2025 Shared Task on Machine Translation Metrics for Indigenous Languages Models score translations from Spanish into Bribri, Guarani, and Nahuatl. The organizers rank systems by Pearson correlation with bilingual speakers’ ratings. | AmericasNLP | Translation | University | Not in cards | Jan 2025 | ||||
| Amharic Speech Emotion Dataset Models identify the emotion expressed in a short Amharic recording. The speakers use four regional dialects. | Ephrem Afele Retta and colleagues | Audio understanding and voice | Everyday | Not in cards | Nov 2021 | ||||
| AMI Meeting Corpus Models identify speakers or transcribe English meetings. NVIDIA evaluates both headset mixes and distant microphone recordings. | AMI Consortium | Audio understanding and voice | Everyday | Not in cards | |||||
| AmsterTime Models match historic photographs of Amsterdam with contemporary street views. The benchmark measures recognition of the same place across time. | Burak Yildiz and colleagues | General visual questions | Everyday | Not in cards | Mar 2022 | ||||
| AMUSE Multimodal models reason over multi-speaker audio-visual interactions in zero-shot, guided and agentic modes across six task families. | Apple | Audio understanding and voice | Expert | Not in cards | Dec 2025 | ||||
| AndroidWorld-Knowledge Agents attempt 48 AndroidWorld tasks across five apps. Each app provides guidance for its simplest task, and harder tasks test whether the agent can reuse it. | Beijing Jiaotong University and Alibaba Group | Phones | Everyday | Not in cards | May 2025 | ||||
| ANES presidential vote choice Models use survey profiles to predict which major party candidate each respondent backed in three US presidential elections. Researchers measure agreement with reported votes. | Lisa P. Argyle and colleagues | Social simulation | Everyday | Not in cards | Sep 2021 | ||||
| Animal Kingdom Models identify actions in wildlife videos that can show several behaviors at once. The benchmark also tests pose estimation and locating events described in text. | Xun Long Ng and colleagues | Video understanding | School | Not in cards | Apr 2022 | ||||
| Animal-Bench Models answer 41,839 questions about animal videos. The overall score averages accuracy across 13 tasks, including action prediction and recognizing predator and prey behavior. | Beijing University of Posts and Telecommunications and China Telecom | Video understanding | School | Not in cards | |||||
| Animals with Attributes 2 Models use shared attributes to recognize animal classes they did not see during training. The benchmark provides 37,322 photographs across 50 classes with 85 attributes. | Institute of Science and Technology Austria and collaborators | General visual questions | School | Not in cards | Jun 2017 | ||||
| ANLI R1 Models label premise and hypothesis pairs from the first adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Facebook AI Research (Nie et al.) | Language reasoning | Everyday | Not in cards | Oct 2019 | ||||
| ANLI R2 Models label premise and hypothesis pairs from the second adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Facebook AI Research (Nie et al.) | Language reasoning | Everyday | Not in cards | Oct 2019 | ||||
| ANLI R3 Models label premise and hypothesis pairs from the third adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Facebook AI Research (Nie et al.) | Language reasoning | Everyday | Not in cards | Oct 2019 | ||||
| AnthroBench Models hold multi-turn conversations that test 14 anthropomorphic behaviors. Automated classifiers and human participants evaluate the generated dialogue. | Google DeepMind | Instructions and chat | Everyday | Not in cards | Feb 2025 | ||||
| Anthropic cyber safeguard evaluations Classifier recall on violative cyber exchanges, block rates on defensive vulnerability-finding and secure-coding work, and the rewind-attacker robustness test. | Anthropic | Refusing harmful requests | Not in cards | ||||||
| Anthropic harmful-request and wellbeing evaluations Single-turn harmful and benign request sets, multi-turn tests, child safety, suicide/self-harm, disordered eating and election integrity. | Anthropic | Refusing harmful requests | Not in cards | ||||||
| Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. | Anthropic (with Axiom Bio and others) | Research science | Expert | Not in cards | 2026 | ||||
| Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. | Anthropic Frontier Red Team | Finding and exploiting real bugs | Expert | Not in cards | Oct 2026 | ||||
| Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. | Anthropic | Refusing harmful requests | Not in cards | ||||||
| Anthropic model-written evals LM-generated datasets testing persona, sycophancy and advanced-AI-risk behaviors. | Anthropic | Deception and alignment | Not in cards | Dec 2022 | |||||
| Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. | Anthropic | Deception and alignment | Not in cards | ||||||
| Apollo Research coding agent monitor calibration Models rate the severity of problems in coding sessions. Apollo compares the ratings with reference labels across three datasets. | Apollo Research | Deception and alignment | Expert | Not in cards | Jul 2026 | ||||
| AppAgent Test Set Agents follow 45 instructions across nine Android apps, including scheduling email and changing music playback. Tasks time out after ten steps. | Tencent | Phones | Everyday | Not in cards | Dec 2023 | ||||
| Applied Theory of Mind (Apollo Research) Agents complete tasks that require tracking what another participant knows. Apollo measures success at three difficulty levels. | Apollo Research | Deception and alignment | Everyday | Not in cards | Aug 2024 | ||||
| AppWorld Agents write code against 457 APIs of 9 simulated everyday apps to finish 750 tasks for about 100 fictional users. State-based unit tests check the outcome and collateral damage. | Stony Brook University and Allen Institute for AI | Apps and MCP servers | Everyday | Not in cards | Jul 2024 | ||||
| AQuA Two LLM research loops (factor discovery, model development) reuse evidence from their own past experiments inside sealed sandboxes. Scored by IC and held-out Sharpe. | Princeton University (Mengdi Wang) et al. | AI research tasks | Research frontier | Not in cards | Aug 2026 | ||||
| AQUA-RAT Models solve algebraic word problems and choose one of five answer options. Evaluation compares the selected option with the answer key, while reference rationales explain the solution. | Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom at DeepMind | Word problems | School | Not in cards | May 2017 | ||||
| AraBench Models translate dialectal Arabic into English. The suite covers travel conversations, media transcripts and religious texts. Researchers score translations against references with BLEU. | Qatar Computing Research Institute | Translation | University | Not in cards | Dec 2020 | ||||
| ArAIEval 2024 Models mark persuasive passages in Arabic text and name the technique. They also classify Arabic memes using the image, its extracted text, or both. | ArAIEval organizers | Reading in many languages | University | Not in cards | Mar 2024 | ||||
| ArAIEval 2023 Replaced in Mar 2024 | Not in cards | Jun 2023 | |||||||
| AraSentEval 2026 Models classify the sentiment of Arabic hotel reviews. They also rewrite a sentence to express the opposite sentiment while keeping its topic. | AraSentEval organizers | Reading in many languages | Everyday | Not in cards | Dec 2025 | ||||
| ARC Challenge MT Models answer science questions from ARC-Challenge in translation. The harness scores answers in the target language against the original choices. | LumiOpen | Reasoning in many languages | School | Not in cards | May 2024 | ||||
| Architectural Code Review v1 Models review code changes with planted architectural defects. The evaluator measures precision and recall separately on six complete diffs and 15 single-defect excerpts. | korck | Fixing real repositories | Expert | Not in cards | Aug 2026 | ||||
| ArguAna Counterargs Models retrieve a counterargument for a given argument. The evaluation checks where the annotated counterargument appears in the ranking. | Henning Wachsmuth and colleagues | Reading comprehension | School | Not in cards | |||||
| Aria Digital Twin Models infer the layout of objects from first-person images. The SAM 3D Objects evaluation compares predicted positions and orientations with reference scene geometry. | Meta | Spatial reasoning and counting | School | Not in cards | Jun 2023 | ||||
| ARPA Armenian Paraphrase Corpus Models decide whether two Armenian sentences express the same meaning. | Arthur Malajyan and colleagues | Reading in many languages | School | Not in cards | Sep 2020 | ||||
| ArtCap Models associate paintings with descriptions of their contents. Google compares image encoders by how often the matching caption appears in their top five. | ArtCap team | General visual questions | Everyday | Not in cards | Nov 2022 | ||||
| Arti4D Models watch first-person recordings of people opening furniture. They predict whether parts rotate or slide and estimate the axis of motion. | University of Freiburg and collaborators | Spatial reasoning and counting | School | Not in cards | Sep 2025 | ||||
| Articulate3D Models locate movable parts and handles in scanned rooms. They predict how each part moves, including its axis and the location of its hinge. | INSAIT and ETH Zurich | Spatial reasoning and counting | School | Not in cards | Dec 2024 | ||||
| Artificial Analysis Decision Index Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning. | Artificial Analysis | Composite indices | Everyday | Not in cards | Oct 2026 | ||||
| Artificial Analysis Progress Index Artificial Analysis's index that tracks how model capability changes over time on one continuous scale, launched alongside Intelligence Index v5. | Artificial Analysis | Composite indices | Not in cards | Oct 2026 | |||||
| Artificial Analysis Search Index One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores. | Artificial Analysis | Deep research | Expert | Not in cards | Aug 2026 | ||||
| ArXivCS Models answer research-level computer science questions derived from recent arXiv papers, each with a reference final answer. The first set has 46 questions from August 2026. | MathArena (ETH Zurich SRI Lab) | Scientific code | Research frontier | Not in cards | Sep 2026 | ||||
| ArXivPhys Models answer research-level physics questions derived from recent arXiv papers, each with a reference final answer. | MathArena (ETH Zurich SRI Lab) | Research science | Research frontier | Not in cards | Sep 2026 | ||||
| ASCEND Models transcribe conversations that switch between Mandarin and English. The evaluation accounts for Chinese characters and English words. | Holy Lovenia and colleagues | Speech recognition and translation | Everyday | Not in cards | Dec 2021 | ||||
| ASDiv Models solve English arithmetic word problems with varied phrasing. The reference answers identify the expected numerical result. | Shen-Yun Miao and colleagues | Word problems | School | Not in cards | Jul 2020 | ||||
| ASFX-SED Models locate sound effects in audio recordings. Meta reports area under the receiver operating characteristic curve for event localization. | Yusong Wu and colleagues | Audio understanding and voice | Everyday | Not in cards | Jul 2025 | ||||
| Asian Language Treebank Models translate parallel sentences across Asian languages. The llm-jp-eval harness uses the parallel corpus for machine translation. | NICT, UCSY, and ALT project partners | Translation | School | Not in cards | |||||
| ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. | Google DeepMind | Jailbreaks and prompt injection | Everyday | Not in cards | Nov 2026 | ||||
| ASK Norwegian Grammatical Error Correction Models correct grammatical errors in Norwegian learner text. The harness compares the edits with annotated corrections. | Matias Jentoft, University of Oslo | Reading in many languages | School | Not in cards | |||||
| Asper-Bench Agents turn complex user requests into programs that execute multi-step actions in a simulated digital assistant environment. Automated checks compare the resulting state with the requested goal. | Apple | General assistants | Expert | Not in cards | Jul 2025 | ||||
| Assembly101-O Models watch people assemble toy vehicles and flag the first procedural mistake. Training uses correct procedures, and evaluation stops at the first mistake in an incorrect sequence. | Sapienza University of Rome and University of Catania | Video understanding | Everyday | Not in cards | Apr 2024 | ||||
| AssertionForge design evaluation Models generate SystemVerilog assertions from design specifications and source code. Cadence Jasper counts proven assertions and measures their coverage. | NVIDIA | Electronics | Expert | Not in cards | Mar 2025 | ||||
| ASSIN 2 Models classify entailment and rate semantic similarity for new pairs of Brazilian Portuguese sentences. The entailment task has two classes. | Livy Real, Hugo Gonçalo Oliveira, and Erick Fonseca | Reasoning in many languages | School | Not in cards | Jun 2019 | ||||
| ASSIN Original Replaced in Jun 2019 | Erick Fonseca and colleagues | Not in cards | Jul 2016 | ||||||
| AssistantBench Web agents do 214 realistic, time-consuming tasks on the open web, such as monitoring real-estate markets. Answers are checked automatically against gold answers. | Tel Aviv University, Allen Institute for AI and others | Web browsing | Everyday | Not in cards | Jul 2024 | ||||
| AssurAI Models respond to Korean prompts across 35 risk categories. Model judges rate the safety of text, image, audio, and video outputs from 1 to 5. | Telecommunications Technology Association and KAIST with collaborators | Deception and alignment | Everyday | Not in cards | Nov 2025 | ||||
| ASTRA-bench Agents use personal context and interactive tools to complete more than 2,000 multi-step scenarios. Automated checks score whether each user intent is satisfied. | Apple | General assistants | Expert | Not in cards | Oct 2026 | ||||
| Astro-QA Models answer astronomy questions in English and Chinese. The authors combine answer key checks and text similarity scores, weighting questions by type and difficulty. | Guizhou University and collaborators | Science questions | University | Not in cards | Jun 2024 | ||||
| AstroCLIMB Models compare pairs of astronomy figures or captions. They determine whether the pair shares a figure, shares a paper, comes from citation-linked papers, or comes from unrelated papers. | Center for Astrophysics, Harvard & Smithsonian, and Oak Ridge National Laboratory | Lab and data work | University | Not in cards | Jul 2026 | ||||
| AstroLLaVA image and text retrieval Models match astronomical images with descriptions. Google evaluates whether an image retrieves the matching description among its top five results. | S. Zaman and colleagues | General visual questions | University | Not in cards | Apr 2025 | ||||
| AstroMemBench Spacecraft agents use information from earlier observations to complete simulated missions. The environment scores task completion, safety, and resource use. | You Wu and colleagues | Embodied reasoning | University | Not in cards | Nov 2026 | ||||
| ATEPP Models identify pianists from the notes, timing, and dynamics of their performances. The original evaluation includes a mixed repertoire and three composer-specific tests. | Queen Mary University of London | Audio understanding and voice | Expert | Not in cards | May 2022 | ||||
| ATLAS (Exa) Agents search the web to build a full table of every entity that meets a set of conditions, with 2 to 10 attributes each. A row counts only if every cell is correct. | Exa | Deep research | Expert | Not in cards | Oct 2026 | ||||
| Audio PAIRS Models answer spoken questions adapted from PAIRS. The evaluation checks whether changing demographic cues changes their choices. | UCSC VLAA and the AHELM team | Bias, fairness and politics | Everyday | Not in cards | Nov 2024 | ||||
| Audio Reasoning Challenge 2026 Models answer questions about speech, music, and environmental audio and explain their reasoning. Incorrect answers score zero. A model judge grades correct answers for reasoning quality, with the middle three of five evaluation runs averaged. | Audio Reasoning Challenge organizers, led by Shanghai Jiao Tong University and collaborators | Audio understanding and voice | Everyday | Not in cards | Nov 2025 | ||||
| Audio-NTREX-4L Speech translation models translate long-form French, Spanish, Portuguese and German audio into English. | Kyutai | Translation | Expert | Not in cards | Feb 2026 | ||||
| Audio-Visual Event Localization Models identify an event that can be both seen and heard in a video and locate when it occurs. | University of Rochester | Video understanding | Everyday | Not in cards | Mar 2018 | ||||
| AudioBench Audio language models follow instructions about speech, voice and environmental sounds. The paper covers 8 tasks and 26 datasets, including speech recognition, spoken question answering, voice traits and audio scenes. The repository has since grown to more than 50 datasets. | AudioLLMs team (Bin Wang and coauthors) | Audio understanding and voice | Everyday | Not in cards | Jun 2024 | ||||
| AudioCaps Models match natural language descriptions to audio clips. Meta also compares how accurately encoders match audio, video, and text from the same clip. | Chris Dongjoo Kim and colleagues | Audio understanding and voice | Everyday | Not in cards | |||||
| AudioMNIST Models identify a spoken digit from a recording. The test includes recordings from different speakers. | Sören Becker and colleagues | Speech recognition and translation | Everyday | Not in cards | Jul 2018 | ||||
| AudioSep Eval Benchmark Models isolate a sound from an audio mixture using a caption or a sound label. Scores compare the separated waveform with the original sound. | Xubo Liu and colleagues | Audio understanding and voice | University | Not in cards | Aug 2023 | ||||
| AudioSet Strong Models locate labeled sound events in time. The evaluation uses human annotations of event boundaries in AudioSet clips. | Google Research | Audio understanding and voice | Everyday | Not in cards | |||||
| AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. | Tencent | Tasks from real jobs | University | Not in cards | Nov 2026 | ||||
| Augmented Reality Patent Search Agents search 12,000 augmented and virtual reality patent applications to answer technical questions. A model judges each answer. | John Oh | Questions that need search | University | Not in cards | Nov 2025 | ||||
| Australian BAS Reconciliation Models reconcile a synthetic Australian goods and services tax ledger, correct a Business Activity Statement, and identify bookkeeping errors. The grader weighs totals and fault detection. | afzal0 | Finance | Expert | Not in cards | Sep 2026 | ||||
| Autocast Models forecast real events using questions from forecasting tournaments and news available before each prediction date. | Andy Zou and collaborators | Real-world events | Expert | Not in cards | Jun 2022 | ||||
| AutoDS scientific machine learning cases Agents build predictive models for maize yield, polymer glass transition temperature, and supercomputer job outcomes. Each case supplies a train split and a held-out test. | ITMO AutoDS | Machine learning engineering | Expert | Not in cards | Aug 2026 | ||||
| Autonomy-10 Agents complete browser, desktop and tool-use tasks across nine domains that measure skill, flexibility and intelligence. | HUD AI | Web browsing | Expert | Not in cards | Jan 2025 | ||||
| AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. | Politecnico di Torino and collaborators (Gioacchini et al.) | Finding and exploiting real bugs | Expert | Not in cards | Oct 2024 | ||||
| AutoResearch An agent edits a small nanochat training script and runs 5 minute training experiments overnight. Scored by final validation bits per byte, where lower is better. | Andrej Karpathy | AI research tasks | Research frontier | Not in cards | Mar 2026 | ||||
| AvalancheBench (May study) Agents recover underlying patterns from synthetic enterprise data and report their findings. | Snowflake and collaborators | Data science and SQL | Expert | Not in cards | May 2026 | ||||
| AvalonBench Models play Resistance Avalon while discussing, deceiving, deducing, and negotiating with other agents. | AvalonBench authors | Multi-agent games | Everyday | Not in cards | Nov 2023 | ||||
| AVCC Next-Speaker Prediction Models predict who will speak after a pause in a conversation. The authors compare two models with human listeners on the same 1,730 events. | KTH Royal Institute of Technology | Speech recognition and translation | Everyday | Not in cards | Nov 2026 | ||||
| AVPS-360 Models locate and track the person making a sound in panoramic video. The annotations connect each sound source to a person across frames. | Tampere University | Audio understanding and voice | Everyday | Not in cards | Oct 2026 | ||||
| aws-bench Coding agents and model pairs diagnose misconfigurations, provision infrastructure and operate live cloud setups in disposable real AWS accounts. | aws-bench maintainers | Terminal tasks | Expert | Not in cards | Jul 2026 | ||||
| Aya Red-teaming Models answer harmful prompts in eight languages across nine harm categories. Judges score refusal and safety behavior against annotated categories. | Cohere Labs | Refusing harmful requests | Everyday | Not in cards | Jun 2024 | ||||
| Babel-ImageNet Models classify ImageNet images using class descriptions in different languages. The benchmark measures how recognition accuracy changes with the prompt language. | Gregor Geigle, Radu Timofte, and Goran Glavaš | General visual questions | Everyday | Not in cards | Jun 2023 | ||||
| BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. | Qwen Team, Alibaba Group | General assistants | Expert | Not in cards | Sep 2026 | ||||
| bAbI QA Tasks Models answer questions about short synthetic stories. The tasks isolate skills such as tracking objects, combining supporting facts and reasoning about locations. | Jason Weston and colleagues | Language reasoning | School | Not in cards | Feb 2015 | ||||
| BackendBench LLMs write kernels that override PyTorch core operators. PyTorch's own OpInfo and FACTO tests check correctness, and real model tensor shapes measure speed. | Meta (PyTorch) | GPU kernels | Expert | Not in cards | Jun 2025 | ||||
| Backrooms: 3D Clue-Trail Navigation Agents follow visual clues through a first-person maze. Only an escape without a wrong doorway counts as a solve. | Keeno Corp | Spatial reasoning and counting | Everyday | Not in cards | Nov 2026 | ||||
| BackSearch Paid API that searches a frozen web archive as of a given date, for forecasting backtests and reproducible web-enabled evals. A tool, not a benchmark. | General Reasoning | Real-world events | Not in cards | Jul 2026 | |||||
| Backtracing Models identify the sentence that prompted a student's question or a news reader's curiosity. In conversations, they find the turn that prompted an emotional response. | Rose E. Wang and colleagues | Finding facts in long inputs | Everyday | Not in cards | Mar 2024 | ||||
| BanglaSER Models identify the emotion in Bangla sentences recorded on phones and laptops. | Rakesh Kumar Das and colleagues | Audio understanding and voice | Everyday | Not in cards | Mar 2022 | ||||
| BanSpeech Models transcribe Bangladeshi Bangla broadcasts across 13 domains. Word error rate compares predictions with human transcripts. | Shahjalal University of Science and Technology | Speech recognition and translation | Everyday | Not in cards | Oct 2022 | ||||
| BAREC-ST-2026 Models assign Arabic sentences and documents to 19 reading levels. The blind test includes texts from 17 publishers absent from the original corpus. | CAMeL Lab and BAREC organizers | Reading in many languages | University | Not in cards | May 2026 | ||||
| BAREC Shared Task 2025 Replaced in May 2026 | Not in cards | May 2025 | |||||||
| BarterBench Agents trade scarce goods to meet inventory targets. The hub port assigns one model-controlled trader to a marketplace of scripted opponents. | James E. Ball | Multi-agent games | University | Not in cards | Mar 2026 | ||||
| Bash Agent Evals Agents query GitHub issues and pull requests through bash, SQL, or a combination. Braintrust scores answers against the dataset. | Braintrust | Apps and MCP servers | University | Not in cards | Jan 2026 | ||||
| BashBench Agents complete multi-step system administration tasks while red teams attempt covert malicious actions in the same shell environment. | Redwood Research and UK AI Security Institute | Dangerous capabilities | Expert | Not in cards | Jun 2025 | ||||
| BasqueGLUE Models analyze Basque text for tasks including sentiment, topic classification, coreference and entailment. Each task has its own evaluation metric. | Gorka Urbizu and colleagues | Reading in many languages | School | Not in cards | Jun 2022 | ||||
| BBH-ja Models solve Japanese translations of BIG-Bench Hard reasoning problems. | Preferred Networks | Reasoning in many languages | School | Not in cards | Jul 2025 | ||||
| BCSC Self-Assessment Program ophthalmology questions Models answer ophthalmology board preparation questions across subspecialties. Researchers compare their choices with the academy’s answer key. | American Academy of Ophthalmology | Medicine | Expert | Not in cards | Jan 2023 | ||||
| BdSLW401 Models identify one of 401 Bangla Sign Language words in a video. The collection records signs from the front and side. | National University Bangladesh and Islamic University of Technology | Video understanding | Everyday | Not in cards | Dec 2024 | ||||
| BdSLW60 Models identify an isolated Bangla Sign Language word in a video clip. Accuracy measures how often the predicted word matches its label. | National University Bangladesh and Islamic University of Technology | Video understanding | Everyday | Not in cards | Dec 2023 | ||||
| BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors Models judge whether math tutors identify and locate student mistakes and give guidance students can act on. A fifth track asks which tutor wrote each reply. | BEA 2025 shared task organizers | Other professions | University | Not in cards | Mar 2025 | ||||
| Beancount Ledger Agents reconcile bank statements, repair a Beancount ledger, and allocate payments to invoices. The evaluator checks balances and payment records against the underlying transactions. | cangultekn | Documents and spreadsheets | University | Not in cards | Sep 2026 | ||||
| BEANS-Zero Models follow text prompts to classify, detect or caption animal sounds without task-specific training. | Earth Species Project | Audio understanding and voice | University | Not in cards | Nov 2024 | ||||
| BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. | Stanford Vision and Learning Lab | Simulated robots | Everyday | Not in cards | Mar 2024 | ||||
| Behnke and Geiger Flexible Job Shop Instances Agents build schedules or write solvers that assign operations to eligible machines within work centers. The objective is to finish the last job as early as possible. | Dennis Behnke and Martin Josef Geiger | Tasks from real jobs | University | Not in cards | Jan 2012 | ||||
| BelarusianGLUE Models classify Belarusian sentiment, recognize entailment, judge grammar, and distinguish word meanings in context. | Maksim Aparovich and colleagues | Reading in many languages | Everyday | Not in cards | Jul 2025 | ||||
| Ben-10 Regional ASR Models transcribe spontaneous speech in ten regional dialects of Bengali using spellings checked by linguists. The organizers report word and character error by dialect. | Bengali.AI and Islamic University of Technology Computer Society | Speech recognition and translation | Everyday | Not in cards | Apr 2024 | ||||
| bgGLUE Models perform language understanding tasks in Bulgarian, including sentiment analysis on movie reviews and question answering on high school exams. | Momchil Hardalov and colleagues | Reading in many languages | Everyday | Not in cards | Jun 2023 | ||||
| BHS Syntactic Evaluation Models compare sentences in Basque, Hindi and Swahili. The test checks whether the model prefers the grammatical member of each pair. | Daria Kryvosheieva and colleagues | Reasoning in many languages | Everyday | Not in cards | Jan 2025 | ||||
| Bible Accuracy Benchmark Models quote verses from a named Bible translation. They earn points for accurate quotations and lose points for inventing verses. | YouVersion | Hallucination rates | School | Not in cards | Jul 2026 | ||||
| BIBLE Translation Benchmark (Meta) Models translate verses from the Gospel of John into English. Meta scores the translations against reference verses using chrF++ and xCOMET. | FAIR at Meta | Translation | University | Not in cards | Mar 2026 | ||||
| BIG-bench Language models do over 200 tasks contributed by hundreds of authors, from linguistics and math to social bias. Most tasks are scored by exact match or multiple choice. | Google and 400+ contributing authors | Language reasoning | University | Not in cards | Jun 2022 | ||||
| BigLaw Bench Retrieval Retrieval systems find the most relevant passages in legal documents. Harvey compares retrieval results with attorney annotations. | Harvey | Law | Expert | Not in cards | Nov 2024 | ||||
| BigLaw Bench Workflows: SPA Deal Points Models extract deal terms from stock purchase agreements. Harvey measures accuracy against terms extracted and checked by attorneys. | Harvey | Law | Expert | Not in cards | Sep 2024 | ||||
| BigLaw Bench: Global Models perform legal work specific to the UK, Australia and Spain. Local legal experts designed tasks and grading criteria for each jurisdiction. | Harvey and Mercor | Law | Expert | Not in cards | Feb 2026 | ||||
| BigLaw Bench: Research Agents use search tools to find US case law and give cited answers. The tasks span corporate law, litigation and other practice areas. | Harvey and Snorkel AI | Law | Expert | Not in cards | Mar 2026 | ||||
| BillSum Models summarize legislative bills. The dataset pairs bill text with summaries written for the US Congress and the California legislature. | Anastassia Kornilova and Vladimir Eidelman | Law | University | Not in cards | Nov 2019 | ||||
| Binary Reasoning v1 Models answer machine-checked questions about compiled software across processor architectures. The hub port supplies 100 of 10,251 tasks and a separate set of 29 interactive tasks. | S3RL4 | Fixing real repositories | Expert | Not in cards | Oct 2026 | ||||
| Bing Chat election answers in Switzerland, Bavaria, and Hesse Bing Chat answers questions about candidates, polls, and voting in three elections. Annotators identify factual errors, evasions, and accurate answers. | AlgorithmWatch and AI Forensics | Long-form factuality | Everyday | Not in cards | Oct 2023 | ||||
| Bingo (Visual Hallucination) Models answer questions about images. The benchmark tests whether prior expectations or interference from other visual information cause false answers. | Chenhang Cui and colleagues | Hallucination rates | Everyday | Not in cards | Nov 2023 | ||||
| Bio-Literature Search Level 2 Agents use search tools to compare abstracts from PubMedQA, including their sections, subject headings, and conclusions. Five task types require retrieving or combining evidence. | d42me | Questions that need search | University | Not in cards | Feb 2026 | ||||
| BioASQ Systems answer biomedical questions written by experts, using PubMed articles as evidence. Questions are yes/no, factoid, list or summary types. A second task asks systems to index new PubMed articles with MeSH headings. | BioASQ challenge organizers | Medicine | Expert | Not in cards | 2013 | ||||
| BioCircuit Agents identify groups of features in Qwen3-4B that control a behavior or restore a damaged output. The grader tests their interventions on held-out prompts. | Amit Prakash and Lalithadithya N | AI research tasks | Research frontier | Not in cards | Jul 2026 | ||||
| BioCreative V Chemical-Disease Relation Task Models identify biomedical entities and chemical-induced disease relations in PubMed abstracts. The OpenMed port extracts chemical and disease names. | BioCreative V CDR organizers | Genomics and life sciences | University | Not in cards | |||||
| BioCreative VII LitCovid Models assign one or more of seven research topics to COVID-19 articles using titles and abstracts. | BioCreative and the US National Library of Medicine | Genomics and life sciences | University | Not in cards | Jun 2021 | ||||
| BioCreative VIII Task 3 Genetic Phenotype Normalization from Dysmorphology Physical Examinations Models identify abnormal physical findings in pediatric exam notes and return Human Phenotype Ontology identifiers. A second task also scores the locations of the findings. | Cedars-Sinai Medical Center and Children's Hospital of Philadelphia | Medicine | University | Not in cards | Apr 2023 | ||||
| BiodivNER Models identify and label phrases in biodiversity abstracts and dataset descriptions. Annotators distinguish six kinds of entities, including organisms, locations, and environments. | Nora Abdelmageed and colleagues | Lab and data work | University | Not in cards | Apr 2022 | ||||
| Biological Wastewater Treatment Long-Form QA Models answer 286 closed-book questions about biological wastewater treatment. Three experts grade the answers for relevance and factual accuracy. | Princeton University and collaborators | Science questions | University | Not in cards | Jul 2024 | ||||
| BioML-bench Agents build and evaluate machine learning systems for biomedical experiments spanning proteins, drugs, single cells, imaging, and biomarkers. Task-specific metrics compare submissions with experimental reference results. | Science Machine | Machine learning engineering | Expert | Not in cards | Sep 2025 | ||||
| Biomni-Eval1 Agents use biomedical data and tools to answer 433 questions across ten task types. Each task checks a structured answer against a reference. | Stanford Biomni | Genomics and life sciences | Expert | Not in cards | Oct 2025 | ||||
| BioNNE 2024 Models mark the spans and types of overlapping medical terms in English and Russian abstracts. Organizers score span matches using macro F1. | BioASQ BioNNE organizers | Medicine | University | Not in cards | Feb 2024 | ||||
| BioNNE-L Models link nested terms in English and Russian biomedical abstracts to medical concept identifiers. Organizers report top-one accuracy, top-five accuracy, and mean reciprocal rank. | BioASQ BioNNE-L organizers | Medicine | University | Not in cards | Feb 2025 | ||||
| BioNNE-R Models identify relations between supplied medical entity mentions in English and Russian abstracts. Organizers rank systems by macro F1 across 14 relation types. | BioASQ BioNNE-R organizers | Medicine | University | Not in cards | Jan 2026 | ||||
| BioRED Models extract biomedical relations from 600 PubMed abstracts. The hub port presents entity pairs and asks models to classify their relationship. | National Center for Biotechnology Information | Genomics and life sciences | Expert | Not in cards | Jul 2022 | ||||
| BioRiskEval Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir. | Scale AI | Hazardous knowledge | Expert | Not in cards | Oct 2025 | ||||
| BioTIER Safeguarded model configurations receive harmful and benign biology requests. Scored on refusing requests designated for refusal and allowing benign ones. | SecureBio | Refusing harmful requests | Expert | Not in cards | Mar 2026 | ||||
| BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. | Risa Shinoda and colleagues | Genomics and life sciences | University | Not in cards | Mar 2026 | ||||
| BirdSet Models identify bird species from audio recordings across multiple evaluation datasets. | Lukas Rauch and colleagues | Audio understanding and voice | Expert | Not in cards | Mar 2024 | ||||
| Birdsnap Models identify the North American bird species shown in a photograph. | Thomas Berg and colleagues | Detection and captioning | University | Not in cards | Jun 2014 | ||||
| Bisbee ANES feeling thermometers Models rate social and political groups from a survey respondent’s perspective on a 0 to 100 scale. Researchers compare ratings with the respondent’s answers. | James Bisbee, Joshua Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer Larson | Social simulation | Everyday | Not in cards | May 2023 | ||||
| BixBench-Hypothesis Agents analyze existing datasets to test 51 hypotheses. A rubric awards points for intermediate results and the final conclusion. | Edison Scientific | Lab and data work | Expert | Not in cards | Mar 2026 | ||||
| Blackjack Basic Strategy Models play blackjack hands or select a single move from a dealt position. The grader measures the expected value of each decision against basic strategy. | petern | Puzzles and chess | Everyday | Not in cards | Sep 2025 | ||||
| BLiMP Language models must assign higher probability to the grammatical sentence in each of 67,000 minimal pairs covering English syntax, morphology and semantics. | New York University (Warstadt et al.) | Commonsense | Everyday | Not in cards | Dec 2019 | ||||
| Blind Spot (Carlos Toledo) Models find an input pair that distinguishes a modified circuit from its original or declare the circuits equivalent. Simulation and SAT proofs check the answers. | carlos-toledo | Electronics | Expert | Not in cards | Sep 2026 | ||||
| Blind-Spots-Bench Models answer 235 text and visual problems chosen to expose failures on tasks that people find simple. A Gemini 3 Flash grader compares each response with a structured reference solution and gives binary credit. The grader can execute code to check exact constraints. | École Polytechnique Fédérale de Lausanne | Abstract puzzles and logic | Everyday | Not in cards | Jul 2026 | ||||
| Blobfish Litigation Bench Agents complete 100 tasks in a simulated litigation practice. The grader compares the changes agents make to case records against required outcomes. | Blobfish AI | Law | Expert | Not in cards | Aug 2026 | ||||
| BlueprintSymVL Models use an example symbol to find matching components in piping and instrumentation diagrams. They must give both the correct count and the component labels. | Eindhoven University of Technology, Chalmers University of Technology, and collaborators | Documents, charts and text in images | School | Not in cards | Oct 2025 | ||||
| BM-Bench Image editing models perform instruction-guided non-rigid image manipulations on the ByteMorph benchmark dataset. | ByteDance Seed | Image generation and editing | Expert | Not in cards | Jun 2025 | ||||
| Bornholm Parallel Models match Bornholmsk sentences to their Danish translations by comparing sentence vectors. The test contains 500 sentence pairs. | Scandinavian Embedding Benchmark team | Translation | School | Not in cards | Jul 2023 | ||||
| Bot Analysis Arena Models write trading rules for simulated US stock accounts. Some agents revise their rules after trades. Software tests each revision against the current rules before adoption. | AI Trading Competition | Finance | University | Not in cards | Jul 2026 | ||||
| Break the Grader Models compare a numerical grader with a certified interval and determine whether the grader can accept an incorrect value. Exact certificates verify the answer. | carlos-toledo | Written proofs | University | Not in cards | Sep 2026 | ||||
| Brennan Center election conspiracy chatbot audit Chatbots answer questions about election claims and respond to skeptical follow-ups. Researchers check factual errors, unsupported citations, and whether responses endorse the claims. | Brennan Center for Justice | Long-form factuality | Everyday | Not in cards | Aug 2026 | ||||
| Brennan Center election image identification Chatbots inspect 14 synthetic election images in English and Spanish. Researchers count how many images each chatbot identifies as synthetic in each language. | Brennan Center for Justice | Detection and captioning | Everyday | Not in cards | Aug 2026 | ||||
| BRFauna Models answer questions, simplify sentences, and summarize documents in Brazilian Portuguese. The suite compares generated text with references across three datasets. | Gabriel Assis, Cláudia Freitas, and Aline Paes | Reading in many languages | University | Not in cards | Jun 2025 | ||||
| BRIGHT Retrieval systems find documents for queries from StackExchange, LeetCode and math competitions where relevance needs reasoning. Scored by nDCG@10. | Su et al. (HKU and collaborators) | Questions that need search | Expert | Not in cards | Jul 2024 | ||||
| BrowseComp Long Context 256k Models answer BrowseComp questions from a context of 128K to 256K tokens built from the needed web pages and related distractor pages. Scored by accuracy. | OpenAI | Reasoning over long documents | Expert | Not in cards | Aug 2025 | ||||
| BSNLP 2019 Shared Task Models find, classify, normalize, and link named entities across Slavic languages. | Jakub Piskorski and colleagues | Reading in many languages | School | Not in cards | Aug 2019 | ||||
| BUCC 2018 bitext mining shared task Models find translation pairs in two collections of sentences. The evaluation compares the retrieved pairs with a reference list. | Pierre Zweigenbaum and colleagues | Translation | School | Not in cards | |||||
| bun-server-bench Agents implement 50 backend services in Bun. Hidden tests grade behavior such as transaction handling, concurrent requests, and rate limits. | Tiny Computer AI | Building whole projects | Expert | Not in cards | Jun 2026 | ||||
| BURST Models recognize, segment, and track objects across video frames. The benchmark tests how well they associate the same object across time. | Ali Athar and colleagues | Video understanding | Everyday | Not in cards | Sep 2022 | ||||
| BVA Case Citation Dataset Models recommend the next legal citation in a veterans’ appeal decision. The authors measure how often the recorded citation appears among the top recommendations. | Stanford RegLab and Technical University of Munich | Law | Expert | Not in cards | May 2021 | ||||
| caBreu Models summarize Catalan news articles. Reference summaries cover extractive, abstractive, and very short summaries. | Barcelona Supercomputing Center | Reading in many languages | School | Not in cards | Sep 2023 | ||||
| CACAE 2025 Models label the role of each sentence in a Chinese essay and identify relations between groups of sentences. Relations account for 60 percent of the score. | East China Normal University | Reading in many languages | University | Not in cards | Feb 2025 | ||||
| AMCAE 2024 Replaced in Feb 2025 | School | Not in cards | Mar 2024 | ||||||
| CADECv2 Models extract adverse drug reactions and drug names from patient reviews. The corpus covers 26 drugs across four groups. | CSIRO, Emory University, and the University of Sydney | Medicine | University | Not in cards | May 2024 | ||||
| CADEC v1 Replaced in May 2024 | CSIRO | Not in cards | Dec 2014 | ||||||
| CADmium Fusion 360 Reconstruction Models reconstruct CAD parts from text descriptions of Fusion 360 designs. The authors measure shape distance and count invalid outputs. | Prashant Govindarajan and colleagues | CAD models | University | Not in cards | Jul 2025 | ||||
| CADPrompt Models write CadQuery programs from text descriptions of mechanical parts. The authors align each generated shape with its reference and measure point cloud distance. | Kamel Alrashedy and colleagues | CAD models | University | Not in cards | Oct 2024 | ||||
| CALLHOME American English Speech Models transcribe spontaneous telephone conversations in American English and assign words to speakers. NVIDIA uses the CH109 test subset. | Linguistic Data Consortium | Speech recognition and translation | Everyday | Not in cards | |||||
| CALLHOME diarization Models identify speakers in multilingual telephone conversations. NVIDIA evaluates Part 2 with two to six speakers per call. | NIST and Linguistic Data Consortium | Audio understanding and voice | Everyday | Not in cards | |||||
| Caltech-101 Models identify object categories in photographs. Meta compares the classification accuracy of image encoders on this collection. | California Institute of Technology | General visual questions | Everyday | Not in cards | |||||
| Caltech-UCSD Birds 200 Models identify bird species from photographs. The benchmark distinguishes species with similar appearance. | California Institute of Technology and University of California San Diego | General visual questions | University | Not in cards | Sep 2010 | ||||
| CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. | University of Freiburg | Simulated robots | Everyday | Not in cards | Dec 2021 | ||||
| CAMeL Models fill gaps in Arabic sentences with names, foods, and other cultural references. The Cultural Bias Score measures how often models favor Western over Arab alternatives. | Georgia Institute of Technology | Bias, fairness and politics | Everyday | Not in cards | Nov 2023 | ||||
| Can I Play It? Models assign a difficulty grade from one to nine to each piano score. The original study reports balanced accuracy across the nine grades. | Universitat Pompeu Fabra and Sogang University | Audio understanding and voice | University | Not in cards | May 2022 | ||||
| capa Feature Identification Models identify capabilities in disassembled functions from the EMBER2024-capa dataset. The grader compares their feature lists with reference labels and gives partial credit. | kyleavery | Secure code and review | Expert | Not in cards | Sep 2025 | ||||
| CAPBENCH-English Models assign a policy topic to English bills, legislative speeches, and news stories. The Easy tier scores agreement with four unanimous human coders. | US Policy Agendas Project and poltextLAB | Reading comprehension | University | Not in cards | Oct 2026 | ||||
| CapQuiz Models are evaluated through human-verified questions answered from generated video captions, measuring factuality and coverage of salient content. | Apple | Video captioning | Expert | Not in cards | Sep 2026 | ||||
| CareManage-Bench Models respond to patient scenarios involving medication safety, care plans, and triage. The harness checks the responses against case specific criteria. | Habib Ur Rehman | Medicine | University | Not in cards | Oct 2026 | ||||
| Carena-Hard-Auto Models answer Chinese prompts. A judge compares pairs of answers, and the evaluation uses a Bayesian ranking model. | Beijing Academy of Artificial Intelligence | Chat and instructions in many languages | Everyday | Not in cards | |||||
| CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. | sinatras | Simulated robots | Everyday | Not in cards | Feb 2026 | ||||
| CaseHOLD Models read a passage from a US court decision with the holding of a cited case masked out and pick the correct holding from five choices. The dataset has more than 53,000 questions drawn from US case law. | Stanford RegLab | Law | Expert | Not in cards | Apr 2021 | ||||
| Casual Conversations v2 Models classify attributes of speakers in recorded interviews. The HELM audio tasks compare age and gender predictions with participants’ own labels. | Bilal Porgali and colleagues | Audio understanding and voice | Everyday | Not in cards | Mar 2023 | ||||
| CatalanQA Models extract answers from Catalan passages drawn from VilaQuAD and ViquiQuAD. The dataset balances question types. | Barcelona Supercomputing Center | Reading in many languages | School | Not in cards | Jun 2022 | ||||
| Catalonia Independence Corpus Models identify an author's stance on Catalan independence from posts in Catalan and Spanish. | Elena Zotova and colleagues | Reading in many languages | School | Not in cards | May 2020 | ||||
| CatCoLA Models classify Catalan sentences by whether a native speaker would consider them grammatical. | Universitat Pompeu Fabra and Barcelona Supercomputing Center | Reading in many languages | School | Not in cards | May 2024 | ||||
| Cattle Trade Agents play a long economic game with auctions, hidden offers, bargaining, bluffing, and resource limits. | Cattle Trade authors | Multi-agent games | Everyday | Not in cards | May 2026 | ||||
| CauGym Models infer causal effects from descriptions and probabilities. Test questions vary the wording, add irrelevant facts, or remove information needed to answer. | Shanghai Artificial Intelligence Laboratory, Fudan University and Tongji University | Language reasoning | University | Not in cards | Feb 2026 | ||||
| CausalReasoningEnv Models identify how to estimate a causal effect from a graph and specify the necessary probability queries. The dataset contains 100 evaluation graphs. | irfanjamil | Language reasoning | University | Not in cards | Mar 2026 | ||||
| CDVQA Models compare two aerial images and answer questions about which land cover changed and by how much. The two tests use the same 968 image pairs with different answer distributions. They contain 39,686 and 31,036 questions, respectively. | Zhenghang Yuan, Lichao Mou, Zhitong Xiong, and Xiao Xiang Zhu | General visual questions | School | Not in cards | Dec 2021 | ||||
| CEFR Adherence Benchmark Models answer English learners at a requested CEFR level. The evaluation scores the reply's relevance and whether its language fits the target level. | Astra-K | Following precise instructions | University | Not in cards | Oct 2026 | ||||
| Cekura Bench Models complete separately scored speech recognition, speech generation, voice-agent, real-call, turn-taking and voice-quality benchmarks. Cekura publishes no combined score. | Cekura | Audio understanding and voice | Everyday | Not in cards | Jul 2026 | ||||
| Cekura Production Voice Agents Published composite score for production voice agents. | Cekura | Audio understanding and voice | Everyday | Not in cards | Jul 2026 | ||||
| Censorship & Propaganda Eval Models answer 803 prompts about censored topics and official narratives. A judge scores factual coverage and endorsement of propaganda separately. | Nous Research | Bias, fairness and politics | University | Not in cards | Nov 2026 | ||||
| chABSA Models identify the target and sentiment of opinions in the business results sections of Japanese annual reports. | Chakki | Reading in many languages | University | Not in cards | Dec 2017 | ||||
| Channel Switching Eval Models respond to benign policy conflicts with and without monitoring cues. The evaluation measures how those cues affect behavior and stated reasoning. | anthone | Deception and alignment | University | Not in cards | Jun 2026 | ||||
| CHART-Info 2024 Models classify chart images, identify text and axes, and recover plotted values. The authors compare baselines on five of the seven tasks. | CHART-Info team | Documents, charts and text in images | School | Not in cards | Sep 2024 | ||||
| ChatGPT Chemistry Assistant MOF synthesis extraction Models extract 11 synthesis details from papers about metal-organic frameworks. Researchers check the reported materials, amounts, temperatures, and reaction times against their annotations. | University of California, Berkeley | Lab and data work | University | Not in cards | May 2023 | ||||
| ChCatExt Models reconstruct headings and paragraphs from Chinese bid notices, financial announcements, and credit rating reports. The combined test contains 65 documents. | Soochow University and Huawei Cloud | Reasoning over long documents | School | Not in cards | Apr 2023 | ||||
| ChemCoTBench Models interpret and edit molecular structures, optimize molecular properties and predict reaction outcomes. Each task has its own scoring rule. | Peking University, IDEA and Yale University | Science questions | University | Not in cards | May 2025 | ||||
| ChemProt Models identify the relationship between a chemical and a protein in a biomedical abstract. The shared task evaluates five relationship types. | BioCreative VI organizers | Genomics and life sciences | Expert | Not in cards | |||||
| Chess Notation Evaluation Models read chess positions in Forsyth-Edwards Notation and describe the board in JSON. The scorer checks fields against the expected position. | apokryphosx | Puzzles and chess | School | Not in cards | Sep 2025 | ||||
| Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. | IBM Research and MIT (Wu et al.) | Medicine | Expert | Not in cards | Jul 2021 | ||||
| ChestX-ray14 Models read frontal chest X-rays from the NIH Clinical Center and say whether thoracic conditions are present. | NIH Clinical Center (Wang et al.) | Medicine | Expert | Not in cards | May 2017 | ||||
| CheXpert Models read chest radiographs and label up to 14 observations. The test labels come from board-certified radiologists. | Stanford ML Group (Irvin, Rajpurkar et al.) | Medicine | Expert | Not in cards | Jan 2019 | ||||
| CHiME-6 Track 2: Unsegmented Speech Recognition Models identify who spoke when and transcribe dinner party recordings without utterance boundaries. The organizers rank systems by word error rate after matching speakers. | CHiME Challenge organizers | Speech recognition and translation | Everyday | Not in cards | Dec 2019 | ||||
| Chinese Classical Poetry Appreciation 2026 Models explain words, lines, and allusions in classical Chinese poems. They also complete analogies and choose among interpretations. | Harbin Institute of Technology, Shenzhen, and the Chinese Information Processing Society of China | Reading in many languages | University | Not in cards | Feb 2026 | ||||
| Chinese Classical Poetry Appreciation 2025 Replaced in Feb 2026 | Not in cards | Jan 2025 | |||||||
| Chinese Essay Discourse Coherence Corpus Models grade coherence in Chinese school essays, identify topic sentences, and classify relationships between passages. The authors hold out the final 100 essays. | East China Normal University and Microsoft Research Asia | Reading in many languages | School | Not in cards | Dec 2023 | ||||
| Chinese Essay Rhetoric Recognition Evaluation 2025 Models find figures of speech across sentences in Chinese student essays. They classify eight figures and extract the words that form four of them. | East China Normal University and Shanghai Linguan Data Technology | Reading in many languages | School | Not in cards | Dec 2024 | ||||
| Chinese Essay Rhetoric Recognition and Understanding 2024 Replaced in Dec 2024 | East China Normal University and Microsoft Research Asia | Not in cards | Jan 2024 | ||||||
| Chinese Language Generation Evaluation (CLGE) Models generate Chinese paper titles from scientific abstracts. The evaluation compares the generated titles with references using ROUGE and BLEU. | SuperCLUE | Reading in many languages | University | Not in cards | Dec 2019 | ||||
| Chinese Linguistics and Cognition Challenge Models answer open questions about Chinese language and cognition. Human graders or the publisher’s judge model assess the answers. | Beijing Academy of Artificial Intelligence | Reasoning in many languages | University | Not in cards | |||||
| Chinese Literature Named Entity Recognition and Relation Extraction Models mark named entities in Chinese literary passages and identify relationships between them. The authors score both tasks with F1. | Peking University | Reading in many languages | School | Not in cards | Nov 2017 | ||||
| Chinese Machine Reading Comprehension 2019 Models restore missing sentences in Chinese stories. Some options come from elsewhere in the same story. The organizers score individual blanks and entirely correct passages. | Harbin Institute of Technology and iFLYTEK Research | Reading in many languages | School | Not in cards | May 2019 | ||||
| Chinese Machine Reading Comprehension 2017 Replaced in May 2019 | Not in cards | Apr 2017 | |||||||
| Chinese Student English Essay Models score English essays from Beijing high school exams. Teachers assign up to 8 points for content, 8 for language, and 4 for structure. | Changrong Xiao and colleagues at Tsinghua University and partner institutions | Other professions | University | Not in cards | Jan 2024 | ||||
| Chinese WPLC Models predict the final word in a Chinese novel excerpt. Readers helped select passages where the final sentence alone gave too little information. | Tianjin University and Peng Cheng Laboratory | Reading in many languages | School | Not in cards | Apr 2021 | ||||
| ChiPBench (chip placement) Agents place circuit blocks in chip designs. OpenROAD measures timing, power, and area after routing. Higher timing slack is better. | MIRA Lab and colleagues | Electronics | Expert | Not in cards | Jul 2024 | ||||
| ChronoMagic-Bench Video models generate time-lapse scenes from 1,649 prompts covering biological, human-made, weather, and physical changes. Automatic metrics measure the amount of change and temporal coherence, with a separate GPT-4o assessment of change. | Peking University, Peng Cheng Laboratory, Rabbitpre Intelligence and others | Video generation | Everyday | Not in cards | Jun 2024 | ||||
| Chronos Benchmark II Models forecast future values across 27 time series datasets. The test compares point and probabilistic forecasts against a seasonal baseline. | Amazon and Chronos collaborators | Real-world events | University | Not in cards | Mar 2024 | ||||
| CIFAR-10 Models classify small color images into ten object categories. | University of Toronto | General visual questions | Everyday | Not in cards | Apr 2009 | ||||
| CIFAR-100 Models classify small color images into 100 object categories. | University of Toronto | General visual questions | School | Not in cards | Apr 2009 | ||||
| CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. | Microsoft Research | Spatial reasoning and counting | Expert | Not in cards | Oct 2026 | ||||
| CiteBench Models answer information-seeking questions with citations from retrieved passages. Human annotations score citation quality across relevance, support and completeness. | Amazon Science | Faithful to sources | University | Not in cards | Jun 2025 | ||||
| Cityscapes Models assign a class to each pixel in street scenes filmed from a vehicle. The benchmark compares predicted segments with annotated road scenes. | Cityscapes team | Detection and captioning | Everyday | Not in cards | Apr 2016 | ||||
| CivilComments-WILDS Models classify the toxicity of online comments. The benchmark evaluates how accuracy varies across groups mentioned in the text. | Pang Wei Koh and colleagues | Bias, fairness and politics | Everyday | Not in cards | Dec 2020 | ||||
| ClaimBench Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency. | Vals AI | Finance | University | Not in cards | Oct 2026 | ||||
| CLARA-MeD Spanish Medical Text Simplification Models rewrite Spanish medical sentences for patients. The collection includes 3,800 aligned pairs and 1,200 clinical trial sentences that linguists simplified. | CLARA-MeD project | Medicine | University | Not in cards | May 2022 | ||||
| ClassEval Models generate whole Python classes for 100 hand-built tasks. Tests check each method and the class as a whole. | Fudan University | Function-level problems | University | Not in cards | Aug 2023 | ||||
| Classical Chinese Historical Event Detection Models locate words that signal events in the Twenty-Four Histories. They classify each event under nine broad categories and 67 specific types. | Beijing Language and Culture University | Reading in many languages | University | Not in cards | Aug 2023 | ||||
| CLEAR (Clinical Information Extraction) Models read patient notes and classify a medical condition as present, absent, or uncertain. Human reviewers supply the reference labels. | Ivan Lopez and colleagues | Medicine | Expert | Not in cards | Jan 2025 | ||||
| CLEAR French Medical Text Simplification Models simplify French sentences from drug information, medical encyclopedias, and research reviews. The authors released 100 test pairs. | University of Lille | Medicine | University | Not in cards | Dec 2020 | ||||
| ClearHarm Models answer unambiguously harmful CBRN questions. Judges score whether attacks elicit harmful responses from the model. | FAR.AI | Jailbreaks and prompt injection | Expert | Not in cards | Jun 2025 | ||||
| CLEF 2026 SimpleText Task 1 Models rewrite sentences or whole Cochrane abstracts for the public. The organizers compared outputs with plain-language summaries from 175 new systematic reviews. | University of Amsterdam and University of Western Brittany | Medicine | University | Not in cards | Sep 2025 | ||||
| CLEF eHealth 2019 Technology-Assisted Reviews Task 2 Models rank the articles returned by a systematic review search so reviewers can screen relevant titles and abstracts first. | CLEF eHealth | Medicine | University | Not in cards | Apr 2019 | ||||
| CLEF HIPE 2026 Models decide whether a historical text places a person at a named location. Organizers rank systems by balanced accuracy across three labels. | CLEF HIPE organizers | Reasoning in many languages | University | Not in cards | Sep 2025 | ||||
| CLEF HIPE 2022 Replaced in Sep 2025 | Reading in many languages | Not in cards | Sep 2021 | ||||||
| CLEF JOKER 2023 Task 2: Pun Location and Interpretation Models identify the punning word in English, French, or Spanish jokes. The English interpretation test also asks for words that express both meanings. | CLEF JOKER organizers | Reasoning in many languages | School | Not in cards | Sep 2022 | ||||
| CLEF JOKER 2024 Task 2: Humour Classification According to Genre and Technique Models assign one of six humor techniques to an English text. Organizers rank runs by accuracy and report F1 for each technique. | CLEF JOKER organizers | Reading comprehension | School | Not in cards | Sep 2023 | ||||
| CLEF JOKER 2025 Task 1: Humour-Aware Information Retrieval Systems retrieve texts that match a query and contain humor. Organizers report mean average precision separately for English and European Portuguese. | CLEF JOKER organizers | Questions that need search | Everyday | Not in cards | Oct 2024 | ||||
| CLEF JOKER 2024 Task 1: Humour-Aware Information Retrieval Replaced in Oct 2024 | Not in cards | Sep 2023 | |||||||
| CLEF JOKER 2025 Task 3: Onomastic Wordplay Translation Models translate English names containing wordplay into French using a description. The organizers check exact matches and have translators assess alternatives. | CLEF JOKER organizers | Translation | University | Not in cards | Oct 2024 | ||||
| CLEF JOKER 2026 Task 1: Humour-Aware Information Retrieval Systems retrieve humorous passages that match a topic in English or Hinglish. Organizers rank systems by mean average precision. | CLEF JOKER organizers | Questions that need search | Everyday | Not in cards | Oct 2025 | ||||
| CLEF JOKER 2026 Task 2: Pun Translation Models translate English puns into French. The automatic score checks whether a translation contains an annotated French word or phrase that carries the pun. | CLEF JOKER organizers | Translation | University | Not in cards | Oct 2025 | ||||
| Click Calibrate Models click a circle or a named word in generated screenshots. The evaluation checks accuracy across image sizes and coordinate formats. | Prime Intellect | Finding things on screen | Everyday | Not in cards | May 2026 | ||||
| Climate intervention forecasts for beliefs and policy support Models predict how 11 interventions change climate beliefs and policy support. Researchers compare predicted effects with observed effects across both outcomes. | Madalina Vlasceanu, Kimberly Doell, and the International Collaboration to Understand Climate Action | Social simulation | University | Not in cards | Nov 2023 | ||||
| ClimateCheck 2026 Models retrieve five abstracts for a climate claim and judge the evidence in each. They also label which disinformation narratives, if any, the claim uses. | ClimateCheck organizers (DFKI and colleagues) | Faithful to sources | University | Not in cards | Nov 2025 | ||||
| ClimateCheck 2025 Replaced in Nov 2025 | Not in cards | Feb 2025 | |||||||
| Clinical Abbreviation Sense Inventory Models expand an ambiguous abbreviation using the surrounding words in an English clinical note. Evaluators compare the answer with a clinician's annotation. | University of Minnesota | Medicine | University | Not in cards | Oct 2012 | ||||
| Clinical Laboratory Test Reports Models extract laboratory results from report images into structured JSON. Google compares predictions with reference fields using macro and micro F1. | Esraa Abdelmaksoud, Ahmed Gadallah, and Ahmed Asad | Documents, charts and text in images | School | Not in cards | Nov 2021 | ||||
| CLIP representation bias evaluation Models associate images of everyday objects with gender terms. The evaluation measures how far the associations depart from an equal preference across groups. | Ibrahim Alabdulmohsin and colleagues | Bias, fairness and politics | Everyday | Not in cards | Mar 2024 | ||||
| CLIPSep Sound Separation Models isolate a requested sound from a mixture using a text or image query. The two tests mix music or everyday sounds with interfering audio. | Hao-Wen Dong and colleagues at Sony and UC San Diego | Audio understanding and voice | University | Not in cards | Nov 2022 | ||||
| Clotho Models write a caption describing what can be heard in an audio recording. | Konstantinos Drossos and colleagues | Audio understanding and voice | Everyday | Not in cards | Oct 2019 | ||||
| Clotho OVL-REL Test Set Models predict listener ratings for audio quality and caption relevance. Four generators produced environmental sounds from 100 captions drawn from Clotho. | Minoru Kishi and colleagues | Audio understanding and voice | Everyday | Not in cards | Nov 2025 | ||||
| Cloud Expansion CE80-10 Agents complete a cloud network security exercise. The verifier checks the resulting cloud configuration and observed network behavior. | human-intuition | Secure code and review | Expert | Not in cards | Sep 2026 | ||||
| Cloud Expansion CEExfil-0 Agents complete a cloud security capstone. A verifier reads the environment after the task to assess the outcome. | human-intuition | Capture-the-flag challenges | Expert | Not in cards | Sep 2026 | ||||
| Cloud Expansion RX00 Agents complete a simulated cloud reconnaissance assessment. The verifier compares the submitted result with the recorded state of the exercise. | human-intuition | Capture-the-flag challenges | Expert | Not in cards | Sep 2026 | ||||
| CLUE Models are fine-tuned or prompted on nine Chinese language understanding tasks, including sentence classification, sentence-pair inference and machine reading comprehension. The score averages the task results. | CLUE benchmark team | Reading in many languages | School | Not in cards | Apr 2020 | ||||
| CLUENER2020 Models identify named entities in Chinese text across ten categories. | CLUE benchmark team | Reading in many languages | School | Not in cards | |||||
| CMD-AD-Eval Models describe what happens on screen during gaps in movie dialogue. The authors compare the descriptions with timed human narration using text overlap and model judgments. | University of Oxford, École des Ponts and Shanghai Jiao Tong University | Video captioning | Everyday | Not in cards | Apr 2024 | ||||
| Cochrane RCT Summarization Models read the titles and abstracts of clinical trials and write the conclusions of the Cochrane review that included those trials. | Byron Wallace, Sayantan Saha, Frank Soboczenski, and Iain Marshall | Medicine | Expert | Not in cards | Aug 2020 | ||||
| Cochrane Simplification Models turn medical review abstracts into plain-language summaries. The original evaluation compares generated text with Cochrane summaries, while the hub port adds a model judge. | Ashwin Devaraj and collaborators | Medicine | Expert | Not in cards | Mar 2021 | ||||
| Cochrane-auto Models simplify sentences, paragraphs, or abstracts from Cochrane reviews. The authors aligned technical text with plain English summaries. | University of Amsterdam | Medicine | University | Not in cards | Nov 2024 | ||||
| COCO-O Models detect objects under natural changes in image appearance. The benchmark compares detection on drawings, cartoons, and other visual domains. | COCO-O authors | Detection and captioning | Everyday | Not in cards | Jul 2023 | ||||
| COCO-Stuff 164K Models label each pixel with an object or background category. The benchmark adds annotations for regions such as sky, grass, and pavement. | Holger Caesar and colleagues | Detection and captioning | Everyday | Not in cards | |||||
| COCOTEROS Models write a Spanish sentence that uses three required words and fits the supplied context. | GPLSI, University of Alicante | Reading in many languages | School | Not in cards | May 2024 | ||||
| Code Review Bench (Martian) AI code review tools review pull requests from five large open-source projects. An LLM judge matches their comments to human-verified golden comments. | Martian | Fixing real repositories | Expert | Not in cards | Feb 2026 | ||||
| CodecMOS-Accent Speech systems reconstruct recordings or clone a voice across ten English accents. Listeners score naturalness, speaker similarity, and accent similarity separately. | Wen-Chin Huang, Nicholas Sanders, and Erica Cooper | Audio understanding and voice | Everyday | Not in cards | Mar 2026 | ||||
| CodeInsights Models solve C++ assignments and reproduce student programming behavior. The tasks compare correctness, mistakes, runtime efficiency, and failures on edge cases. | CodeInsights contributors | Function-level problems | University | Not in cards | Jul 2025 | ||||
| CodeXGLUE A suite of 10 code tasks across 14 datasets, grouped as code to code, text to code, code to text and text to text. Tasks include clone detection, defect detection, code completion, code repair, code search, code generation and code summarization. Each task has its own metric, such as accuracy, BLEU or MRR. | Microsoft Research Asia and Microsoft | Function-level problems | School | Not in cards | Feb 2021 | ||||
| CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. | Sakana AI | Running a business | University | Not in cards | Jun 2026 | ||||
| Cohere North internal evals Cohere internal evals of its North agent platform: agentic question answering, spreadsheet data analysis and memory use. LLM-as-judge scoring. | Cohere | Documents and spreadsheets | University | Not in cards | May 2026 | ||||
| COIN instructional video analysis Models recognize steps in videos of people completing everyday tasks. The benchmark supplies annotated actions and their time intervals. | COIN team | Video understanding | Everyday | Not in cards | Mar 2019 | ||||
| CoIR Models retrieve relevant code or text across programming languages. The suite includes searches from code to code and from descriptions to code. | CoIR team | Function-level problems | University | Not in cards | Jul 2024 | ||||
| ColDeR Models rank passages containing relevant evidence against alternatives that exploit shortcuts such as length, repetition, or word overlap. | Mohsen Fayyaz and colleagues | Questions that need search | School | Not in cards | Mar 2025 | ||||
| COLIEE 2026 Models retrieve Canadian precedents and apply Japanese statutes. The statute retrieval track also requires a correct yes or no answer with sufficient supporting articles. | COLIEE organizing committee | Law | University | Not in cards | Dec 2025 | ||||
| COLIEE 2025 Replaced in Dec 2025 | Not in cards | Jan 2025 | |||||||
| COLIEE 2024 Replaced in Jan 2025 | Not in cards | Oct 2023 | |||||||
| CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. | Alibaba.com (Accio) | Running a business | Expert | Not in cards | Aug 2026 | ||||
| Common Objects in 3D v2 Models estimate camera poses from images of objects viewed from different angles. Meta compares predicted poses with the reference poses. | Meta AI | Spatial reasoning and counting | School | Not in cards | |||||
| Common Syntactic Processes in Text-to-Image Generation Models generate images from sentences with different grammatical structures. Evaluators check whether each image preserves the relationships expressed in the sentence. | Evelina Leivada and colleagues | Image generation and editing | School | Not in cards | Oct 2022 | ||||
| Commonsense Reasoning and Moral Understanding in Children’s Stories Models read Chinese fables and choose answers about events or the moral. Moral questions contribute 60 percent of the final score. | Shanxi University and Hefei University of Technology | Reading comprehension | School | Not in cards | Feb 2024 | ||||
| CompactionBench Agents continue multi-turn coding sessions while context compaction occurs during software evolution tasks. Scores measure task quality and compaction cost. | Microsoft Research | Long terminal sessions | Expert | Not in cards | Oct 2026 | ||||
| Company Enrichment API Benchmark Company data APIs fill profile fields from a website. Code and expert checks score headquarters, employees, founding year, social links, industry, ticker and revenue. | Firecrawl | Apps and MCP servers | Expert | Not in cards | Oct 2026 | ||||
| COMPASS Agents plan travel across transportation, lodging and tickets while satisfying hard constraints and optimizing stated preferences over multiple turns. | Apple | Apps and MCP servers | Expert | Not in cards | Oct 2025 | ||||
| CompassBench v1.3 Models answer Chinese and English tasks in coding, instruction following, knowledge, language, math and reasoning. OpenCompass publishes objective and subjective evaluation configurations. | OpenCompass | Composite indices | University | Not in cards | Jul 2024 | ||||
| ComplaintsClustering Models group consumer complaints by financial product. The score measures whether each group contains one product and whether complaints about that product stay together. | The Hong Kong University of Science and Technology | Finance | Everyday | Not in cards | Jul 2024 | ||||
| ComputeEval Models write CUDA code for handcrafted problems on kernels, runtime APIs and GPU libraries. Held-out test harnesses check correctness, and optional benchmarks time the code. | NVIDIA | Speed and optimization | Expert | Not in cards | Apr 2025 | ||||
| Concurrency Bugs (tanishgudise) Agents repair Python packages with asynchronous or threaded code. Hidden tests reproduce each bug deterministically across 15 repair tasks. | tanishgudise | Fixing real repositories | University | Not in cards | Oct 2026 | ||||
| CoNLL-2002 Named Entity Recognition Models mark entity spans and classify their types in Spanish and Dutch news text. | Erik Tjong Kim Sang | Reading in many languages | School | Not in cards | |||||
| ConRef-STANCE-ita Models infer a user's position on Italy's constitutional referendum from a post, a repost, and a reply. Researchers score support, opposition, and neutrality. | Mirko Lai, Viviana Patti, Giancarlo Ruffo, and Paolo Rosso | Reading in many languages | Everyday | Not in cards | Apr 2018 | ||||
| Contextual Complex Instruction-Following Retrieval Retrievers rank passages under instructions about multiple requirements. | Contextual AI | Following precise instructions | School | Not in cards | Mar 2025 | ||||
| Contextual Invoice OCR Models extract invoice fields using a business’s past bills. Ramp counts invoices where every field matches the submitted bill. | Ramp Labs | Finance | University | Not in cards | Mar 2026 | ||||
| Contextual Recency-Based Instruction-Following Retrieval Retrievers rank passages under instructions about recency. | Contextual AI | Following precise instructions | Everyday | Not in cards | Mar 2025 | ||||
| Contextual Reranker Technical Documentation Benchmark Retrievers rank documents for queries supplied by enterprise customers. | Contextual AI | Questions that need search | Expert | Not in cards | Mar 2025 | ||||
| ContextualBench CRM agents answer enterprise questions against contextual business data and receive accuracy, cost, speed, trust and safety measurements. | Salesforce AI Research | Questions that need search | Expert | Not in cards | Sep 2024 | ||||
| Contract Intelligence Models answer questions about contract terms. Harvey compares answers with attorney reviewed labels. | Harvey | Law | Expert | Not in cards | Nov 2025 | ||||
| Contract Workflows (Applications) Systems complete legal work involving contracts. The benchmark assesses whether each deliverable satisfies its legal and presentation criteria. | Legal Benchmarks | Law | Expert | Not in cards | |||||
| ContractNLI Models decide whether a contract supports, contradicts, or omits a claim. The original benchmark also asks them to identify the passages that justify their decision. | Yuta Koreeda and Christopher D. Manning | Law | Expert | Not in cards | Oct 2021 | ||||
| ConvoMem Models answer questions about earlier conversations, including changed facts and user preferences. The dataset contains 75,336 items across 100 fictional users. | Salesforce Research | Finding facts in long inputs | Everyday | Not in cards | Nov 2025 | ||||
| Cooperative Codenames (Max Bittker) Two models cooperate as cluegiver and guesser to identify target words on a hidden board. The default evaluation uses 200 separately generated boards. | maxbittker | Multi-agent games | Everyday | Not in cards | Mar 2026 | ||||
| COPIOUS biodiversity entity recognition Models mark names of organisms, places, habitats, people, and time expressions in historical biodiversity texts. The test measures how closely the marked phrases match human annotations. | University of Manchester and University of the Philippines Diliman | Lab and data work | University | Not in cards | Jan 2019 | ||||
| CoQA Models answer a series of linked conversational questions about a passage, where later questions depend on earlier turns. Scored by word-level F1. | Stanford University (Reddy, Chen and Manning) | Reading comprehension | Everyday | Not in cards | Aug 2018 | ||||
| CoQCat Models answer follow-up questions in Catalan using a passage and the preceding conversation. | Barcelona Supercomputing Center | Reading in many languages | School | Not in cards | Sep 2023 | ||||
| CORAAL Models transcribe recordings of African American speech. NVIDIA reports long-form word error rates by location. | CORAAL project | Speech recognition and translation | Everyday | Not in cards | |||||
| CORE English web register identification Models label English web documents by register, such as news reporting, discussion, or instructions. A document can have several labels. Researchers report F1. | Veronika Laippala and colleagues | Reading comprehension | School | Not in cards | Aug 2022 | ||||
| Corpus of Multilingual Code-switched Soap Opera Speech Models transcribe South African television dialogue in which speakers switch between English and four Southern African languages. | Stellenbosch University | Speech recognition and translation | Everyday | Not in cards | May 2018 | ||||
| Cortex Analyst Multi-Turn Question Rewriting Models rewrite a follow-up question so it stands alone. Mistral Large 2 judges whether the rewrite preserves the necessary conversation history. | Snowflake | Multi-turn conversation | University | Not in cards | Nov 2024 | ||||
| CortexRetrievalBench Retrieval systems rank financial documents for a practitioner’s question. The benchmark checks whether an authoritative primary source appears at the top. | micro1 | Finding facts in long inputs | Expert | Not in cards | |||||
| CoSHE-Eval Models transcribe 1,985 conversational Hindi and English clips. The references use Devanagari for Hindi and Latin script for English. | Soket Labs | Speech recognition and translation | Everyday | Not in cards | Nov 2025 | ||||
| CounselBench-100 Agents investigate legal matters and decide which actions the evidence supports. The verifier checks decisions, changes to case records, and advice across 100 synthetic tasks. | Blobfish AI | Law | Expert | Not in cards | Aug 2026 | ||||
| Country211 Models predict the country where a photograph was taken. | OpenAI | Spatial reasoning and counting | University | Not in cards | Feb 2021 | ||||
| Coval Voice AI Benchmarks Speech-to-text, text-to-speech and speech-to-speech models from almost 40 providers are scored every 30 minutes on call-like audio with accents, noise and echo. | Coval | Speech recognition and translation | Everyday | Not in cards | Apr 2026 | ||||
| COVID-Dialogue Models read a patient’s questions about COVID-19 and generate a response. Doctor replies from medical consultations serve as references. | UC San Diego AI4H | Medicine | Expert | Not in cards | |||||
| Cozzo AI Receptionist Benchmark (Round 6) Voice agents quote a dental price, confirm appointments, and handle urgent calls. Cozzo checks spoken details and scores responses to interruptions. | Cozzo Labs LLC | Customer service | Everyday | Not in cards | Oct 2026 | ||||
| Cozzo AI Receptionist Benchmark (Round 5) Replaced in Oct 2026 | Not in cards | Oct 2026 | |||||||
| Cozzo AI Receptionist Benchmark (Round 4) Replaced in Oct 2026 | Not in cards | Oct 2026 | |||||||
| CPRet Models retrieve contest solutions, duplicate problems, or full problems from simplified descriptions. Four retrieval tasks score the ranking of relevant results with NDCG@10. | CUHK, Shanghai AI Laboratory, Tsinghua University and others | Competitive programming | Competition | Not in cards | May 2025 | ||||
| CRAG-MM Multimodal retrieval agents answer single-turn and multi-turn questions grounded in images and changing web content. The benchmark includes wearable-style egocentric images. | Meta FAIR and Superintelligence Labs | Questions that need search | Expert | Not in cards | Oct 2025 | ||||
| Crate Bench Models identify an album and its artist from the cover image. The benchmark provides 300 test covers and supports closed-book and web-search modes. | Zanussbaum | General visual questions | Everyday | Not in cards | Mar 2026 | ||||
| CredData Models identify hardcoded credentials in source files. Samsung publishes reviewed labels and obfuscates credential values. OpenAI scores both tokens and exact spans. | Samsung Research | Secure code and review | University | Not in cards | Aug 2021 | ||||
| CREF Models submit forecasts before the target period begins and receive scores after the actual values become available. | Andreas Auer and colleagues | Real-world events | University | Not in cards | Nov 2026 | ||||
| CrisisMMD Models classify disaster photographs and their accompanying posts for humanitarian use. | Firoj Alam and colleagues | General visual questions | School | Not in cards | May 2018 | ||||
| Cross-lingual Sentiment Consistency Analysis for Classical Chinese Literature Models predict emotions in classical Chinese passages and their English translations. The test checks how accurately each prediction captures the passage and how closely the predictions agree. | Xi’an Jiaotong-Liverpool University and Harbin Institute of Technology, Shenzhen | Reading in many languages | University | Not in cards | Feb 2026 | ||||
| Cross-lingual Test Dataset Models retrieve images from captions written in different languages. The benchmark compares retrieval against paired photographs and descriptions. | Pranav Aggarwal and Ajinkya Kale | Reading in many languages | School | Not in cards | Dec 2020 | ||||
| CrossRE Models label relationships between named entities in sentences from six domains. The natural science test includes 400 sentences covering topics such as comets and planets. | Elisa Bassignana and Barbara Plank | Reading comprehension | School | Not in cards | Oct 2022 | ||||
| CrossSum Models read a news article in one language and summarize it in another. The dataset aligns articles and summaries across more than 1,500 language pairs. | Abhik Bhattacharjee and colleagues | Reading in many languages | School | Not in cards | Dec 2021 | ||||
| CS-FLEURS Models transcribe speech that switches languages within sentences. Four test sets cover 113 language pairs. People read one set aloud, and synthesizers generate the others. | Brian Yan and colleagues | Speech recognition and translation | Everyday | Not in cards | Jul 2025 | ||||
| CSATQA Models read Korean exam passages and choose among five answers. The harness aggregates results across six question groups. | HAE-RAE | Reading in many languages | School | Not in cards | Jul 2023 | ||||
| ČSFD Sentiment Models classify the sentiment of movie reviews from the Czech and Slovak Film Database. | Tomáš Brychcín and Ivan Habernal | Reading in many languages | Everyday | Not in cards | Sep 2013 | ||||
| CSL Models generate paper titles and keywords or classify Chinese scientific papers by field. Scoring uses ROUGE-L for titles, Bpref for keywords, and accuracy for classification. | China University of Geosciences (Beijing), Shenzhen University, Tencent AI Lab and others | Reading in many languages | University | Not in cards | Sep 2022 | ||||
| CSR-Bench Agents deploy and run computer-science research repositories with shell, retrieval and web-search tools. Task success measures whether the requested repository workflow completes. | Amazon Science | Fixing real repositories | Expert | Not in cards | Jan 2025 | ||||
| CSTNews Models summarize Brazilian Portuguese news articles individually or combine reports about one event. Human summaries provide references for both tasks. | Núcleo Interinstitucional de Linguística Computacional | Reading in many languages | School | Not in cards | Jun 2008 | ||||
| CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. | Medicine | Expert | Not in cards | Jan 2026 | |||||
| CT-RATE Models read 3D non-contrast chest CT volumes and say whether each of 18 conditions is present. Scored by macro accuracy or F1 against report-derived labels. | University of Zurich and collaborators (Hamamci et al.) | Medicine | Expert | Not in cards | Mar 2024 | ||||
| CTF-Archive Agents solve capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. | US CAISI (NIST) | Capture-the-flag challenges | Expert | Not in cards | Sep 2025 | ||||
| CTI-to-MITRE Models classify a threat description using MITRE ATT&CK technique labels. The HELM scenario presents a set of candidate labels. | Vittorio Orbinato and colleagues | Secure code and review | University | Not in cards | Aug 2022 | ||||
| CUA-SHADE-Arena Monitors review computer-use agent activity in simulated workplace tasks. The evaluation measures how often they flag unsafe behavior at a fixed false positive rate. | Scale AI | Deception and alignment | University | Not in cards | Aug 2025 | ||||
| CUAD Models review commercial contracts and highlight the clauses a lawyer would flag, across 41 categories such as governing law, non-compete and change of control. Legal experts from The Atticus Project wrote more than 13,000 labels over 510 contracts. Scores measure how well the extracted spans match the expert labels. | The Atticus Project and UC Berkeley | Law | Expert | Not in cards | Mar 2021 | ||||
| CubeBench Agents solve Rubik's Cube tasks across symbolic, visual and partially observed settings to measure spatial reasoning and long-horizon state tracking. | Princeton AI Lab | Spatial reasoning and counting | Expert | Not in cards | Dec 2025 | ||||
| CulturaQA Models answer questions about Greek culture, including art, mythology, politics, and everyday life. | IMISLab | Knowledge in many languages | School | Not in cards | Apr 2026 | ||||
| CURE Models retrieve passages that answer clinicians' questions at the point of care. | Clinia and collaborators | Medicine | Expert | Not in cards | Dec 2024 | ||||
| Current-facts set Models answer questions about current facts. Writer evaluates responses as part of its neutrality and safety study. | Writer | Short factual questions | Everyday | Not in cards | Aug 2026 | ||||
| CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. | Meta (PurpleLlama) | Finding and exploiting real bugs | Expert | Not in cards | Jul 2024 | ||||
| CyberSecEval Visual Prompt Injection Models read text and images where the image carries instructions that try to override the task. Scored by how often the injection succeeds. | Meta (PurpleLlama) | Secure code and review | Expert | Not in cards | Jul 2024 | ||||
| CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. | Meta (PurpleLlama) | Capture-the-flag challenges | Expert | Not in cards | Apr 2024 | ||||
| Cybersecurity and Privacy Legal Agent Eval Agents write legal risk memos for five synthetic privacy and incident reporting matters. The grader checks the completed memo and its coverage of legal issues. | narcolepticchicken | Law | Expert | Not in cards | Jul 2026 | ||||
| Cyc3D Models generate 3D assets from images. The evaluator checks mesh quality and measures how much shape and identity drift across repeated rendering and reconstruction. | Liwen Zhang | Image generation and editing | University | Not in cards | Aug 2026 | ||||
| Cyrillic Turkic language identification Models identify the language of text written in Cyrillic. The test includes Turkic languages and Russian. | tatiana-merz | Reading in many languages | School | Not in cards | |||||
| CzechBankQA Models translate questions about banking records into SQL queries using a supplied database schema. | Yifan Mai, Stanford CRFM | Data science and SQL | University | Not in cards | Oct 2024 | ||||
| D2K-Bench Agents turn expert GPU-kernel designs into efficient Triton, CUDA or CUTLASS kernels across 26 tasks. Runtime timing and design-code scores measure implementation quality. | Qwen Team, Alibaba Group | Speed and optimization | Expert | Not in cards | Oct 2026 | ||||
| DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. | Adyen and Hugging Face | Data science and SQL | Expert | Not in cards | Jun 2025 | ||||
| DACSA Models summarize Catalan and Spanish newspaper articles. Editors’ summaries provide the references. | Encarnación Segarra Soriano and colleagues | Reading in many languages | School | Not in cards | Jul 2022 | ||||
| DaLAJ Models judge the linguistic acceptability of Swedish sentences. | Elena Volodina and colleagues | Reading in many languages | School | Not in cards | May 2021 | ||||
| DaLAJ-GED 2.0 Models decide whether a Swedish sentence contains a grammatical error. The test includes learner writing and examples from Swedish coursebooks. | Elena Volodina and Yousuf Ali Mohammed, University of Gothenburg | Reading in many languages | School | Not in cards | Feb 2023 | ||||
| Daleel 2026 Models identify six kinds of argument in Arabic editorials and debates. They also mark the words that express each argument. | Daleel organizers and QatarDebate Center | Reading in many languages | University | Not in cards | May 2026 | ||||
| DaNE Models locate and classify names in Danish sentences drawn from the Danish Universal Dependencies treebank. | Rasmus Hvingelby and colleagues | Reading in many languages | School | Not in cards | May 2020 | ||||
| DanFEVER Models decide whether Danish encyclopedia evidence supports a claim, refutes it, or leaves it unresolved. The dataset contains 6,407 annotated claims. | IT University of Copenhagen | Faithful to sources | School | Not in cards | Apr 2021 | ||||
| Danish Semantic Reasoning Benchmark Models interpret Danish metaphors, compare word senses, and infer relationships between words and statements. | Centre for Language Technology at the University of Copenhagen and collaborators | Reasoning in many languages | School | Not in cards | May 2024 | ||||
| DANSK Models identify named entities across Danish text domains. The authors annotate a subset of Danish Gigaword using 18 entity types. | Kenneth Enevoldsen and colleagues | Reading in many languages | School | Not in cards | Dec 2024 | ||||
| Danske Talemåder Models select the meaning of a Danish idiom from four definitions. | Society for Danish Language and Literature | Knowledge in many languages | Everyday | Not in cards | |||||
| DarijaBench Models translate, summarize and classify sentiment in Moroccan Arabic. A transliteration task also tests conversion between Arabic and Latin scripts. | Guokan Shang and colleagues | Reading in many languages | Everyday | Not in cards | Sep 2024 | ||||
| DarijaHellaSwag Models read a situation in Moroccan Arabic and choose its most plausible continuation. | Guokan Shang and colleagues | Reasoning in many languages | Everyday | Not in cards | Sep 2024 | ||||
| DarijaMMLU Models answer academic multiple-choice questions in Moroccan Arabic. The evaluation adapts MMLU to the dialect. | Guokan Shang and colleagues | Knowledge in many languages | University | Not in cards | Sep 2024 | ||||
| Data Extraction (Applications) Systems extract requested information from legal documents. The benchmark checks whether the resulting work satisfies the task criteria. | Legal Benchmarks | Law | Expert | Not in cards | |||||
| data-eng-bench Agents build and repair dbt models across 103 retail warehouse tasks. The verifier compares materialized tables row by row against a reference solution. | Snowflake Labs | Data science and SQL | University | Not in cards | Aug 2026 | ||||
| DataCLUE Systems improve the training data for Chinese text classification tasks. | CLUE benchmark team | Machine learning engineering | Expert | Not in cards | |||||
| DataComp image encoder evaluation Models trained on different image and text collections perform the same set of downstream tasks. The evaluation compares transfer performance across 37 tasks. | DataComp team | General visual questions | Everyday | Not in cards | Apr 2023 | ||||
| DataSciBench Agents complete open-ended data-science tasks that require data cleaning, analysis, visualization and modeling. Component checks and overall task success score the work. | Z.ai, Tsinghua University and collaborators | Data science and SQL | Expert | Not in cards | Apr 2026 | ||||
| DAVIS 2017 Models track and segment individual objects throughout a video. The evaluation measures region overlap and boundary accuracy. | DAVIS Challenge organizers | Video understanding | Everyday | Not in cards | Apr 2017 | ||||
| DCASE 2024 Language-Queried Audio Source Separation Models isolate the sound named in a text query. Listeners score sound quality and agreement with the query on 100 real recordings with overlapping sounds. | DCASE Task 9 organizers | Audio understanding and voice | University | Not in cards | Mar 2024 | ||||
| DCASE 2024 Sound Scene Synthesis Models generate a four-second sound scene from a text description. Listeners rate how well the foreground and background match the prompt and assess audio quality. | DCASE | Audio understanding and voice | University | Not in cards | Dec 2023 | ||||
| DCASE 2023 Foley Sound Synthesis Replaced in Dec 2023 | Not in cards | Dec 2022 | |||||||
| DCS 2018 Sanskrit Word Segmentation Models split Sanskrit compounds and reverse sound changes at word boundaries. The sentence score counts an answer only when all its words match. | Oliver Hellwig and Sebastian Nehrdich | Reading in many languages | University | Not in cards | Aug 2018 | ||||
| DDIExtraction 2013 Models identify drug names in biomedical text and classify interactions between pairs of drugs. The OpenMed port covers interaction classification. | SemEval 2013 Task 9 organizers | Medicine | University | Not in cards | Jun 2013 | ||||
| DDisCo Models assign one of three coherence ratings to a Danish Reddit post or Wikipedia passage. The test file contains 101 Reddit posts and 100 Wikipedia passages. | Aarhus University and the Alexandra Institute | Reading in many languages | School | Not in cards | Feb 2022 | ||||
| DDXPlus Models diagnose simulated patients from symptoms, clinical history, and demographic information. The OpenMed port asks for the most likely condition among 49 possibilities. | DDXPlus team | Medicine | Expert | Not in cards | May 2022 | ||||
| DealBench-100 Agents perform 100 investment banking workflows across ten synthetic transactions. Deterministic checks grade calculations, decisions, committed changes, and consistency between models and presentations. | Blobfish AI | Finance | Expert | Not in cards | Sep 2026 | ||||
| DEBAGREEMENT Models classify whether a reply agrees with a comment, disagrees, or takes a neutral position. Human labels provide the reference answers. | University of Oxford and Scale AI | Reading comprehension | Everyday | Not in cards | Oct 2021 | ||||
| DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. | Xi'an Jiaotong University | Deception and alignment | Everyday | Not in cards | Oct 2026 | ||||
| Deceptive Reasoning Exposure Suite Monitors inspect model reasoning and answers to detect cases where a harmless looking response conceals deceptive reasoning. | Gray Swan and collaborators | Deception and alignment | Expert | Not in cards | Sep 2025 | ||||
| Deep Memory Retrieval Agents answer one fact question from earlier sessions in a multi-session conversation. A judge compares the answer with the target fact. | MemGPT and Letta | Reading comprehension | Expert | Not in cards | Oct 2023 | ||||
| DeepAmbigQA Search-augmented models answer multi-hop questions that combine entity-name ambiguity with complete answer-set retrieval. | Apple | Questions that need search | University | Not in cards | Nov 2025 | ||||
| Deepgram Voice AI Benchmarks Speech models are compared on word error rate across production audio in 20 languages, while speech synthesis models are compared on read-aloud and hard-prompt sets. | Deepgram | Speech recognition and translation | Expert | Not in cards | Apr 2026 | ||||
| DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. | Google DeepMind | Simulated robots | University | Not in cards | Sep 2024 | ||||
| DeepMind Fast Language Learning Tasks Models learn mappings between unfamiliar words and meanings from short examples. Held-out tasks measure how well they infer new word meanings. | Google DeepMind | Language reasoning | School | Not in cards | Jul 2017 | ||||
| DeepResearchEval Deep research systems answer complex research tasks that require evidence from many web sources. A persona-driven pipeline writes the tasks and keeps only those that need outside search. A model judge scores each report on general and task-specific dimensions, and an agent fact-checks the report's statements with web search even when citations are missing. | Infinity Lab (Shanda Group) and Nanyang Technological University | Deep research | Expert | Not in cards | Jan 2026 | ||||
| DeepResearchEval v1 Research agents prepare reports. The score is average report quality on the original evaluation tasks. | Infinity Lab (Shanda Group) and Nanyang Technological University | Deep research | Expert | Not in cards | Jan 2026 | ||||
| DeepSDO Description Models describe solar features visible in full-disk images from the Solar Dynamics Observatory. The scorer compares generated captions with human descriptions. | Korea Astronomy and Space Science Institute and collaborators | Lab and data work | University | Not in cards | |||||
| Defects4J A collection of 854 real, reproducible bugs from 17 open-source Java projects, each with the buggy and fixed versions and tests that expose the bug. Program repair studies ask a model to fix each bug, and a fix counts when the project's tests pass. It also serves fault localization and test generation research. | University of Washington (René Just and coauthors) | Fixing real repositories | Expert | Not in cards | Jul 2014 | ||||
| DeGennaro Workplace sample Agents act as an employee’s assistant using 80 simulated workplace tools and anonymized company records. Four tasks grade the final state. | doe | General assistants | Expert | Not in cards | Nov 2026 | ||||
| DELEGATE-52 Models perform long delegated document-editing workflows across 52 professional domains. Scores measure artifact fidelity after repeated transformations. | Microsoft Research | Documents and spreadsheets | Expert | Not in cards | Apr 2026 | ||||
| Dementia Family Caregiver Detection Models identify English posts whose authors report having a family member with dementia. Organizers score F1 for that class on 1,769 test posts. | Ari Z. Klein and colleagues | Medicine | Everyday | Not in cards | May 2022 | ||||
| Demosthenes Argument Classification Models identify premises and conclusions in European Union court decisions on fiscal state aid. They also classify premise types and legal reasoning schemes. | Demosthenes team, University of Bologna | Law | Expert | Not in cards | Oct 2022 | ||||
| Describable Textures Dataset Models assign a texture description to a photograph based on how its surface looks. | Mircea Cimpoi and colleagues | General visual questions | Everyday | Not in cards | Nov 2013 | ||||
| DESED Models identify when sound events occur in domestic environments. Meta compares frame-level predictions across audio encoders. | DESED project | Audio understanding and voice | Everyday | Not in cards | |||||
| DesignJanus People rate generated objects from six viewpoints and mark inconsistencies in shape, materials, and identity. The harness compares human ratings with automatic measures. | j-jw-yim | Image generation and editing | Everyday | Not in cards | Mar 2026 | ||||
| Detecting Entities in the Astrophysics Literature Models mark and classify named entities in excerpts from astrophysics papers, including telescopes, astronomical objects, and grants. Organizers score entity spans and token labels. | Center for Astrophysics, Harvard & Smithsonian, and Charles University | Lab and data work | University | Not in cards | May 2022 | ||||
| DETESTS-Dis 2024 Models detect racial stereotypes in Spanish comments and posts and decide whether the stereotypes are implicit. Organizers score majority labels and distributions of annotator judgments. | IberLEF DETESTS-Dis organizers | Reading in many languages | University | Not in cards | Feb 2024 | ||||
| DETOUR Agents ask a simulated user for clues and identify a partially remembered item. | Patronus AI | Questions that need search | Everyday | Not in cards | Feb 2026 | ||||
| Developer Retrieval Benchmark Search systems retrieve repositories, documentation pages and issue or pull request fixes. Deterministic URL matching scores recall and reciprocal rank. | Firecrawl | Apps and MCP servers | Expert | Not in cards | Oct 2026 | ||||
| DevOpsBench-100 Agents resolve 100 operations and reliability tasks in a simulated online store. The verifier compares system state before and after their work. | Blobfish AI | Terminal tasks | Expert | Not in cards | Aug 2026 | ||||
| Deysi Spam Detection Models classify English messages as spam or legitimate. The hub adapter grades the label and response format across 2,725 test messages. | Deysi | Reading comprehension | Everyday | Not in cards | Apr 2023 | ||||
| DGCL Reward Hacking Models decide whether corporate actions comply with Delaware law and provide supporting calculations. Fifty evaluation cases distinguish correct headline decisions from fully correct answers. | narcolepticchicken | Law | Expert | Not in cards | May 2026 | ||||
| DialectSentEval 2026 Models classify sentiment in five Arabic dialects. In a separate task, they rewrite a sentence with the opposite sentiment while preserving its topic and dialect. | DialectSentEval organizers | Reading in many languages | Everyday | Not in cards | May 2026 | ||||
| DiDeMo Models locate the time interval in a video that matches a natural language description. | Lisa Anne Hendricks and colleagues | Video understanding | Everyday | Not in cards | Aug 2017 | ||||
| DigiKlausur Assisted Short Answer Grading Models grade responses to 17 neural network exam questions. One human grader assigned zero, one, or two points to each of 646 entries. | DigiKlausur at Bonn-Rhein-Sieg University of Applied Sciences | Other professions | University | Not in cards | Feb 2019 | ||||
| DIHARD III Models identify who speaks when across eleven recording domains. The benchmark includes overlapping speech. | Ryant and colleagues | Audio understanding and voice | Everyday | Not in cards | Sep 2020 | ||||
| Dillen herbarium specimen label benchmark Models transcribe specimen labels from images of pressed plants. The collection pairs 1,800 images from nine European herbaria with transcribed records. | Meise Botanic Garden and eight partner collections | Documents, charts and text in images | School | Not in cards | Nov 2018 | ||||
| DIODE Models estimate depth in indoor and outdoor scenes. The dataset provides depth measurements from a laser scanner for comparison. | Igor Vasiljevic and colleagues | Spatial reasoning and counting | School | Not in cards | Aug 2019 | ||||
| DIOR Models locate and classify objects in remotely sensed images. The benchmark measures detection accuracy across different object sizes and backgrounds. | Ke Li and colleagues | Detection and captioning | University | Not in cards | Aug 2019 | ||||
| DIOR-RSVG Models draw a box around the object named in a description of an aerial image. The released test split contains 7,500 descriptions. | Northwestern Polytechnical University | Detection and captioning | Everyday | Not in cards | Oct 2022 | ||||
| Dirty Integration Agents collect every record from a paginated API despite unreliable cursors, rate limits, changing field types and timeouts. | buildok | Apps and MCP servers | University | Not in cards | Aug 2026 | ||||
| DISCFinLLMReranking Models rank Chinese news articles for questions about financial events. Each question has two or three relevant articles and ten distractors. | The Hong Kong University of Science and Technology | Finance | University | Not in cards | Jul 2024 | ||||
| Discharge Me! Models draft discharge instructions and brief hospital course summaries from patient records. The shared task compares model output with clinical discharge documentation. | Justin Xu and colleagues | Medicine | Expert | Not in cards | |||||
| DiSCoMaT Models extract material identifiers, constituents, percentages, and units from tables in materials science papers. The original test contains 737 tables from 326 papers. | Indian Institute of Technology Delhi | Lab and data work | University | Not in cards | Jun 2022 | ||||
| DiscoveryBench Agents analyze datasets and produce data-driven hypotheses that evaluators score against structured evidence and domain criteria. | Ai2 and Princeton NLP | AI research tasks | Expert | Not in cards | Jul 2024 | ||||
| Disengagement-Aware Student Simulators Tutors interact with simulated learners who vary in engagement. The authors compare five tutors and validate the student behaviors against real tutoring sessions. | The University of Hong Kong | Other professions | University | Not in cards | Sep 2026 | ||||
| DISPLACE-M 2026 Models identify speakers, transcribe their words, identify medical topics and summarize conversations between community health workers and patients. Each task has its own score. | Indian Institute of Science and DISPLACE-M organizers | Audio understanding and voice | Everyday | Not in cards | Jan 2026 | ||||
| DISPLACE Challenge 2024 Replaced in Jan 2026 | Indian Institute of Science and DISPLACE organizers | Not in cards | Dec 2023 | ||||||
| DISPLACE Challenge 2023 Replaced in Dec 2023 | Indian Institute of Science and National Institute of Technology Karnataka | Not in cards | Dec 2022 | ||||||
| DisTEMIST Models locate disease mentions in Spanish clinical cases and link them to SNOMED CT codes. The organizers score both tasks with microaveraged F1. | Barcelona Supercomputing Center and BioASQ | Medicine | University | Not in cards | Dec 2021 | ||||
| Diving48 Models classify competitive dives from videos by distinguishing their movement sequences. | Yingwei Li and colleagues | Video understanding | Expert | Not in cards | Sep 2018 | ||||
| DOE sample pack Agents complete five simulated workplace assignments in law, private equity, and finance operations. Deterministic checks grade figures and application state, while model judges grade written work. | doe | Tasks from real jobs | Expert | Not in cards | Oct 2026 | ||||
| Dojo Mini Bench Agents complete tasks in LinkedIn, Linear and Gmail through Dojo. Each task has its own completion check. | Chakra Labs | Apps and MCP servers | Everyday | Not in cards | Oct 2025 | ||||
| Dollar Street Models recognize household objects in photos from families at different income levels. The benchmark supports comparisons across regions and income groups. | Gapminder and collaborators | General visual questions | Everyday | Not in cards | |||||
| Dolly Machine Translated v2 Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | Cohere Labs | Chat and instructions in many languages | Everyday | Not in cards | Apr 2026 | ||||
| mDolly v1 Replaced in Apr 2026 | Not in cards | Feb 2024 | |||||||
| Domain Intelligence Benchmark Suite Models extract structured data, call functions, and answer questions using retrieved material. Databricks compares performance on enterprise tasks. | Databricks | Running a business | University | Not in cards | Dec 2024 | ||||
| Dosimeter AI Disinformation Benchmark AI assistants answer questions about contested claims. Dosimeter pools its source checks with published studies that flag citations or repetition of false claims. | Dosimeter | Long-form factuality | Everyday | Not in cards | Oct 2026 | ||||
| Dosimeter US midterm source audit AI assistants answer ten English questions about claims circulating before the US midterms. Dosimeter counts answers that cite a listed source, including refutations. | Dosimeter | Long-form factuality | Everyday | Not in cards | Oct 2026 | ||||
| doteBench Models edit English and Chinese recordings using a transcript that marks where each change belongs. The evaluator checks edit success and preservation outside those regions. | Xiaohongshu and Shanghai Jiao Tong University | Audio understanding and voice | University | Not in cards | Aug 2026 | ||||
| DP-Bench Document parsers and vision models convert PDFs and scanned pages into structured HTML or Markdown. Output is compared with reference structure. | Upstage | Documents, charts and text in images | Everyday | Not in cards | Oct 2024 | ||||
| DravidianLangTech 2026 Hope Speech Detection in Code-Mixed Tulu Models classify the tone of comments that mix Tulu with other languages. A second task distinguishes five kinds of hope or hopelessness. | DravidianLangTech 2026 organizers | Reading in many languages | Everyday | Not in cards | Nov 2025 | ||||
| DravidianLangTech 2026 Multilevel Political Meme Classification in Tamil and Malayalam Models identify a political meme's stance and target from its image and text in Tamil or Malayalam. | DravidianLangTech shared task organizers | General visual questions | Everyday | Not in cards | Nov 2025 | ||||
| DravidianLangTech 2026 Tamil Dialect Speech Recognition and Classification Models identify which of four regional Tamil dialects a speaker uses. A separate track tests how accurately models transcribe the speech. | DravidianLangTech 2026 organizers | Speech recognition and translation | Everyday | Not in cards | Nov 2025 | ||||
| Drawing with LLMs Models generate SVG code from text prompts. Kaggle renders each image and combines visual question answering with an aesthetic score. | Kaggle | Image generation and editing | Expert | Not in cards | Feb 2025 | ||||
| Dreaddit Models classify Reddit posts according to whether their authors express stress. The hub port adds rewards for reasoning and answer format. | Elsbeth Turcan and Kathleen McKeown | Wellbeing and sycophancy | Everyday | Not in cards | Oct 2019 | ||||
| DreamGen Bench Video generation benchmark measuring how well video models adapt to a target robot embodiment. Part of NVIDIA's DreamGen pipeline. | NVIDIA GEAR | Video world models | Not in cards | May 2025 | |||||
| DroidTask Agents select actions for 158 Android tasks across 13 apps. The original completion score requires every action to match the recorded sequence. | Tsinghua University, University of Notre Dame, and Microsoft Research Asia | Phones | Everyday | Not in cards | Aug 2023 | ||||
| DrugProt Models read biomedical abstracts and identify which of 13 labeled relationships connects a marked chemical and protein. | BioCreative VII DrugProt organizers | Genomics and life sciences | University | Not in cards | |||||
| DTU Multi-View Stereo Models reconstruct objects from multiple camera views. The benchmark measures the accuracy and completeness of reconstructed surfaces against scanned reference geometry. | Technical University of Denmark | Spatial reasoning and counting | School | Not in cards | |||||
| Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. | Dual Exam authors | Interactive world models | Everyday | Not in cards | Oct 2026 | ||||
| Dubai-CC Models describe land use changes between two satellite views of Dubai. The Change3D team used 150 of the 500 image pairs for testing. | Genc Hoxha, Seloua Chouaf, Farid Melgani, and Youcef Smara | Detection and captioning | School | Not in cards | |||||
| Duidelijke Taal Models rewrite Dutch text in simpler language. Human-written simplifications provide reference answers. | Instituut voor de Nederlandse Taal | Reading in many languages | School | Not in cards | Dec 2025 | ||||
| DuMateBench Agents complete real-world productivity and office-work tasks in sandboxed workspaces with tools. Task evaluators check the requested artifact and record a reward. | Baidu | Running a business | Expert | Not in cards | Aug 2026 | ||||
| Dutch DPA national voting advice audit Chatbots rank three Dutch political parties for a fictional voter. The regulator checks whether the first recommendation matches the party used to construct the voter’s answers. | Dutch Data Protection Authority | Bias, fairness and politics | Everyday | Not in cards | Oct 2025 | ||||
| Dutch Model Benchmark Models perform Dutch language understanding tasks, including reasoning about the causes and effects of events. | Wietse de Vries and colleagues | Reasoning in many languages | School | Not in cards | Dec 2023 | ||||
| Dynamic Programming (m8ngotree) Models solve generated optimization and counting problems, including knapsack, coin change, and edit distance. A problem-specific algorithm verifies each answer. | m8ngotree | Language reasoning | University | Not in cards | Sep 2025 | ||||
| Dynamic-SUPERB Phase-2 Spoken language models follow text instructions about audio across 180 tasks in speech, music and other sounds. Tasks include classification, regression and sequence generation, and researchers keep contributing new ones. Models run zero-shot, without task-specific training. | Dynamic-SUPERB team (Chien-yu Huang and coauthors) | Audio understanding and voice | Everyday | Not in cards | Nov 2024 | ||||
| E2E-SWE Agents build complete software repositories from natural language specifications in empty workspaces. Hidden tests require every generated project to install and run. | Meta FAIR and Superintelligence Labs | Building whole projects | Research frontier | Not in cards | Sep 2026 | ||||
| Earnings-21 Models transcribe long earnings calls. The benchmark tests recognition of financial terminology in real conversations. | Rev | Speech recognition and translation | Everyday | Not in cards | Apr 2021 | ||||
| Earnings-22 Models transcribe English earnings calls from speakers with different accents. NVIDIA reports long-form word error rate. | Rev | Speech recognition and translation | Everyday | Not in cards | Mar 2022 | ||||
| Earth-Bench-Pro Agents complete Earth observation workflows under three conditions. They follow instructions, plan from supplied observations, or obtain observations at runtime. | OpenDataLab and collaborators | Apps and MCP servers | University | Not in cards | Sep 2026 | ||||
| EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. | Shanghai AI Lab (InternRobotics) | Simulated robots | Everyday | Not in cards | Jun 2026 | ||||
| EBM-NLP Models mark the words that describe participants, interventions, and outcomes in clinical trial abstracts. Medical professionals supplied the test labels. | Northeastern University and collaborators | Medicine | University | Not in cards | Jun 2018 | ||||
| EBM-NLPmod Models extract participants, interventions, comparators, and outcomes from trial titles and methods sections. Annotators separate comparison treatments from interventions. | UTHealth Houston and Yale University | Medicine | University | Not in cards | Aug 2023 | ||||
| ECHR Binary Violation Classification Models read the facts of a European Court of Human Rights case and predict whether the court found a violation. | Ilias Chalkidis and colleagues | Law | Expert | Not in cards | Jun 2019 | ||||
| ECHR Legal Corpus for Argument Mining Models identify argumentative clauses, link related clauses, and distinguish premises from conclusions in European Court of Human Rights rulings. | Prakash Poudyal and colleagues | Law | University | Not in cards | Dec 2020 | ||||
| Economic Dispatch Models assign output levels to generators to meet demand at minimum cost. The grader checks capacity limits, supply balance, and cost against an exact solution. | jwilksbooth | Electronics | Expert | Not in cards | Jul 2026 | ||||
| EDB-Bio Models read experimental evidence and estimate the probability that a scientific hypothesis is true. | Edison Scientific (formerly FutureHouse) | Research science | Expert | Not in cards | |||||
| EDIS Models retrieve news images and their associated text using queries that refer to specific people, places, or events. | Siqi Liu and colleagues | General visual questions | School | Not in cards | May 2023 | ||||
| EduClaw-Bench Agents tutor a simulated learner over 30 days. The benchmark tracks learning gains, responsiveness, helpfulness, and curriculum design across 55 scenarios. | Korea University and collaborators | Other professions | University | Not in cards | Aug 2026 | ||||
| EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. | Aryan Ahmed and colleagues | Other professions | University | Not in cards | Oct 2026 | ||||
| EduFair-Bench Tutors teach the same simulated student under different demographic descriptions. A judge grades individual turns and whole conversations to measure changes in teaching quality. | EPFL | Other professions | University | Not in cards | Sep 2026 | ||||
| Effective FPS Models decide whether video frames are consecutive or duplicated to simulate a higher capture rate. The evaluator checks each prediction against the construction of the clip. | referee-lab | Video understanding | University | Not in cards | Aug 2026 | ||||
| Ego-Exo4D Models estimate human pose during skilled activities filmed from first-person and external viewpoints. Meta reports joint and mesh errors for physical and procedural activities. | Ego-Exo4D consortium | Spatial reasoning and counting | School | Not in cards | Dec 2023 | ||||
| Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. | Yu Dong and colleagues | Simulated robots | University | Not in cards | Nov 2026 | ||||
| Ego4D Goal-Step Models find when a goal or step occurs in first-person video. They also identify the current step or locate a step described in words. | Meta FAIR | Video understanding | Everyday | Not in cards | Dec 2023 | ||||
| Ego4D State Change Object Detection Models locate the object that changes in a first-person video. The official test scores its bounding box in the frame where the change begins. | Ego4D consortium | Video understanding | Everyday | Not in cards | Oct 2021 | ||||
| EgoCross Models answer questions about events in first-person videos. The challenge scores accuracy on 957 four-choice questions across four domains. | Yanjun Li and colleagues | Video understanding | Everyday | Not in cards | Aug 2025 | ||||
| EgoPER Error Detection Models watch first-person cooking videos and flag incorrect steps. The evaluator checks accuracy by step and measures error detection by frame. | Shih-Po Lee and colleagues | Video understanding | Everyday | Not in cards | Dec 2023 | ||||
| EgoProceL Models find recurring steps in first-person videos and mark when each step occurs. The videos show 16 tasks, including cooking and assembling computers. | IIIT Hyderabad and IIT Delhi | Video understanding | Everyday | Not in cards | Jul 2022 | ||||
| EgoToM Video models infer a camera wearer’s goals, beliefs, and future actions from egocentric clips. Multiple-choice questions test theory of mind. | Meta FAIR and Superintelligence Labs | General visual questions | University | Not in cards | Mar 2025 | ||||
| EgyMMLU Models answer multiple-choice subject questions in Egyptian Arabic. The evaluation tests knowledge in a dialect underrepresented in standard Arabic exams. | Abdellah El Mekki and colleagues | Knowledge in many languages | University | Not in cards | May 2025 | ||||
| EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. | Medicine | Expert | Not in cards | Jul 2025 | |||||
| EHRSHOT Models predict clinical outcomes from longitudinal electronic health records with few labeled examples for each task. | Michael Wornow and colleagues | Medicine | Expert | Not in cards | Jul 2023 | ||||
| EHRSQL Models convert clinical questions into SQL queries over electronic health records. The benchmark includes questions that the database cannot answer. | Gyubok Lee and colleagues | Data science and SQL | University | Not in cards | Jan 2023 | ||||
| ELCardioCC 2025 Models find clinical mentions in Greek cardiology discharge letters, link them to ICD-10 codes, and assign codes to each letter with supporting phrases. | Aristotle University of Thessaloniki and collaborators | Medicine | Expert | Not in cards | Feb 2025 | ||||
| Election-Day AI Poisoning in Germany AI search services answer ten German questions about documented political claims. The audit counts answers citing domains on lists of Russian state media or the Pravda network. | Dosimeter | Faithful to sources | Everyday | Not in cards | Sep 2026 | ||||
| Electoral Hallucinations AI services answer questions about three Scottish Parliament constituencies. Two reviewers check responses against election facts and report the percentage that are entirely accurate. | Demos | Long-form factuality | Everyday | Not in cards | May 2026 | ||||
| ELLIPSE Holistic Essay Scoring Models assign one overall English proficiency score to each essay on a five-point scale. The reference is the overall human grade. | Scott Crossley and colleagues | Other professions | University | Not in cards | Feb 2023 | ||||
| Embodied Agent Arena Vision-language agents solve geometry, spatial reasoning, affordance, planning, and manipulation tasks. The 1,000-case evaluation combines 32 established sources with GeoProbe. | HKUST Guangzhou, CUHK, and KnowinAI | Embodied reasoning | Everyday | Not in cards | Oct 2026 | ||||
| EmbodiedCLUE-VLA RoboTwin Evaluation Robot policies perform ten two-handed manipulation tasks under randomized RoboTwin 2.0 Hard conditions. The score averages task success rates over twenty episodes per task. | SuperCLUE | Simulated robots | Everyday | Not in cards | Aug 2026 | ||||
| EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. | Rajat Bhattacharjya and colleagues | Simulated robots | University | Not in cards | Nov 2026 | ||||
| EMDB Models reconstruct human pose and body shape in everyday scenes. The benchmark compares joint positions with electromagnetic measurements. | ETH Zurich | Spatial reasoning and counting | School | Not in cards | Aug 2023 | ||||
| EmpatheticDialogues Models reply to a person describing an emotional experience. Human conversations provide reference responses for evaluating empathy and relevance. | Hannah Rashkin and colleagues | Multi-turn conversation | Everyday | Not in cards | Nov 2018 | ||||
| Employment Law Legal Agent Eval Agents review employment case files and write risk memos. The grader checks whether the memos identify planted legal issues and avoid unsupported findings. | narcolepticchicken | Law | Expert | Not in cards | Jul 2026 | ||||
| EncQA Vision-language models answer chart questions that isolate visual encoding channels and analytic operations. | Apple | Documents, charts and text in images | University | Not in cards | Aug 2025 | ||||
| Encyclopedic VQA Models answer questions about detailed properties of pictured objects and places. The Gemini Embedding 2 report measures whether they retrieve the correct answer. | Google Research | General visual questions | Everyday | Not in cards | Jun 2023 | ||||
| EndoVis 2018 Robotic Scene Segmentation Models segment structures and surgical instruments in video from robot-assisted operations. The evaluation compares predicted masks with reference labels. | EndoVis challenge organizers | Video understanding | Everyday | Not in cards | |||||
| EndoVis18-VQA Models answer questions about organs, tools, tool positions, and interactions in images from robotic kidney surgery. The benchmark accepts class labels or sentence answers in separate tasks. | National University of Singapore and collaborators | Medicine | Expert | Not in cards | Jun 2022 | ||||
| ENEM Challenge Models answer multiple-choice questions from Brazil’s national high school examination in Portuguese. | Eduardo Garcia and contributors | Knowledge in many languages | School | Not in cards | Jan 2024 | ||||
| Enterprise PII Models respond to 100 prompts that request sensitive business information. The evaluation checks whether the responses disclose it. | Patronus AI and Hugging Face | Deception and alignment | University | Not in cards | Jan 2024 | ||||
| Enterprise Scenarios Creative Writing Models write stories in response to 100 prompts. A model grades how engaging the stories are. | Patronus AI and Hugging Face | Creative writing | Everyday | Not in cards | Jan 2024 | ||||
| Enterprise Scenarios Customer Support Dialogue Models answer customer support questions using product information and conversation history. A judge scores their responses. | Patronus AI and Hugging Face | Multi-turn conversation | Everyday | Not in cards | Jan 2024 | ||||
| Enterprise Scenarios Leaderboard Models complete six tasks drawn from workplace settings. The leaderboard combines their scores across all six tasks. | Patronus AI and Hugging Face | Running a business | University | Not in cards | Jan 2024 | ||||
| Enterprise Scenarios Toxicity Models respond to 100 prompts. The Perspective API scores how often their responses contain toxic language. | Patronus AI and Hugging Face | Deception and alignment | Everyday | Not in cards | Jan 2024 | ||||
| Enterprise Task Pareto Frontier (Glean) Models complete 1,000 workplace tasks. A model compares paired answers for correctness, task completion, and readiness for use. Glean estimates preference ratings and cost per task. | Glean | Tasks from real jobs | University | Not in cards | Aug 2026 | ||||
| Enterprise Vendor Eval Agents review vendor documents, question an internal IT team, and recommend whether to approve a purchase. The grader checks their decision and identified requirements. | musa8026 | Running a business | University | Not in cards | Jul 2026 | ||||
| Entity-Deduction Arena Models ask questions to identify a hidden entity in multi-turn games. Success, question count and a combined score measure clarification, state tracking and planning. | Apple | General assistants | Everyday | Not in cards | Feb 2024 | ||||
| EPIC-KITCHENS VISOR Hand Object Segmentation Models outline each hand and the object it touches in a kitchen image. They also identify which hand appears and whether it touches anything. | University of Bristol, University of Michigan, and University of Toronto | Detection and captioning | Everyday | Not in cards | Aug 2022 | ||||
| EPIC-KITCHENS VISOR Semi-Supervised Video Object Segmentation Models receive masks for hands and objects in the first frame of a kitchen video. They track those objects through later frames, including after occlusion. | University of Bristol, University of Michigan, and University of Toronto | Video understanding | Everyday | Not in cards | Aug 2022 | ||||
| EPIC-KITCHENS-100 action anticipation Models predict a person's next action from first-person kitchen video. The benchmark scores predicted verbs, nouns, and their combinations. | EPIC-KITCHENS consortium | Video understanding | Everyday | Not in cards | Jun 2020 | ||||
| Epic-tent-O Models watch someone pitch a tent and flag procedural mistakes using only the video so far. The test includes 15 videos. | Sapienza University of Rome and University of Catania | Video understanding | Everyday | Not in cards | Apr 2024 | ||||
| EPRSTMT Models classify Chinese product reviews as positive or negative after learning from a small labeled sample. Accuracy measures agreement with the sentiment labels. | CLUE benchmark team | Reading in many languages | Everyday | Not in cards | May 2021 | ||||
| ERPBench-100 Agents execute 100 business workflows across ten simulated enterprise resource planning systems. Checks grade decisions, calculations, committed state, and handoffs on a 100-point scale. | Blobfish AI | Apps and MCP servers | Expert | Not in cards | Sep 2026 | ||||
| ERRnews broadcast summarization Models summarize Estonian radio news from automatic transcripts. TalTech compares each summary with the broadcaster’s written summary using ROUGE. | Tallinn University of Technology | Reading in many languages | School | Not in cards | Nov 2022 | ||||
| ESC-50 Models identify an environmental sound from a short recording. | Karol J. Piczak | Audio understanding and voice | Everyday | Not in cards | Oct 2015 | ||||
| Escalation Router Models predict whether Qwen2.5-VL 7B correctly identifies visible hands or grippers in a clip. The reference label comes from two observed runs per clip. | referee-lab | Video understanding | University | Not in cards | Aug 2026 | ||||
| ESCI Shopping Queries Dataset Models rank products for a shopping query and distinguish exact matches, substitutes, complements, and irrelevant items. | Amazon | Questions that need search | Everyday | Not in cards | Jun 2022 | ||||
| EsCoLA Models judge the grammatical acceptability of Spanish sentences taken from linguistic publications. | Universitat Pompeu Fabra and Barcelona Supercomputing Center | Reading in many languages | School | Not in cards | May 2024 | ||||
| EstNER Models identify named entities in manually annotated Estonian news and social media text. | Kairit Sirts | Reading in many languages | School | Not in cards | May 2023 | ||||
| ETH3D Models reconstruct scene geometry from images. Meta evaluates depth predictions against reference measurements on the ETH3D test scenes. | Thomas Schöps and colleagues | Spatial reasoning and counting | School | Not in cards | |||||
| ether0 benchmark Models answer chemistry problems from several task types. The benchmark checks their answers with automated reward functions. | Edison Scientific (formerly FutureHouse) | Research science | Expert | Not in cards | Jun 2025 | ||||
| ETHICS Models predict common moral judgments about text scenarios covering justice, well-being, duties, virtues and commonsense morality. | UC Berkeley (Hendrycks et al.) | Deception and alignment | Everyday | Not in cards | Aug 2020 | ||||
| ETVA Text-to-video systems are scored on fine-grained prompt alignment through generated atomic questions and video-language answers. | Apple | Video generation | Expert | Not in cards | Mar 2025 | ||||
| EUR-Lex-Sum Models summarize European Union legislation in the language of the source document. Official summaries provide references across EU languages. | Dennis Aumiller and colleagues | Law | University | Not in cards | Oct 2022 | ||||
| European Parliament 2010 State of the Union sentiment validation Models rate the sentiment of passages from the European Parliament’s 2010 State of the Union debate. The original evaluation correlates paragraph scores with human ratings. | Sven-Oliver Proksch, Will Lowe, Jens Wäckerle, and Stuart Soroka | Reading comprehension | Everyday | Not in cards | Aug 2018 | ||||
| European Parliament election chatbot audit Chatbots answer ten European election questions in ten languages. Reviewers score procedural answers for accuracy and political answers for neutrality. | Democracy Reporting International | Long-form factuality | Everyday | Not in cards | Apr 2024 | ||||
| EvaCun 2025 Models identify dictionary forms and restore masked words in Akkadian and Sumerian tablet texts. Organizers score accuracy on held-out tablets. | EvaCun organizers | Reading in many languages | Expert | Not in cards | Oct 2024 | ||||
| EvaHan 2026 Models transcribe printed and handwritten Chinese books and locate page elements such as illustrations and annotations. | EvaHan organizers | Documents, charts and text in images | University | Not in cards | Nov 2025 | ||||
| EvaHan 2025 Replaced in Nov 2025 | Reading in many languages | Not in cards | Sep 2024 | ||||||
| EvaHan 2024 Replaced in Sep 2024 | Reading in many languages | Not in cards | Dec 2023 | ||||||
| EvaHan 2023 Replaced in Dec 2023 | Translation | Not in cards | Nov 2022 | ||||||
| EvalConvoLearn Models simulate learners in tutoring conversations. The evaluator compares their answers, mistakes, questions, and turn lengths with patterns in real student conversations. | Baptiste Moreau-Pernet | Other professions | University | Not in cards | Jun 2026 | ||||
| EvalCrafter Text-to-video models generate clips for 700 prompts drawn from real user requests. The clips are scored with 17 automatic metrics for visual quality, content quality, motion and text alignment. Weights fitted to human ratings combine the metrics into one score. | Tencent AI Lab and City University of Hong Kong | Video generation | Everyday | Not in cards | Oct 2023 | ||||
| EvaLearn Models solve 648 problems in 182 sequences to measure learning from feedback across six task types. | ByteDance Seed | Task length | University | Not in cards | Jun 2025 | ||||
| Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment Chatbots respond to 30 questions about suicide. Researchers record how often each chatbot answers directly, grouped by five risk levels that clinicians assigned to the questions. | RAND and Harvard Medical School collaborators | Wellbeing and sycophancy | Expert | Not in cards | Aug 2025 | ||||
| Evaluation of Essay On-Topic Graded Comments Models rate how closely each Chinese school essay follows its prompt and explain their rating. Written feedback supplies 60 percent of the total score. | East China Normal University | Reading in many languages | University | Not in cards | Feb 2025 | ||||
| Event-based Segmentation Dataset Models segment objects in cluttered indoor scenes filmed from a moving robot arm. The benchmark provides reference masks for comparison. | Xiaoqian Huang and colleagues | Video understanding | Everyday | Not in cards | Feb 2023 | ||||
| Evidence Inference 2.0 Models classify the direction of a treatment effect in a trial report and identify text that supports their answer. | Northeastern University and King's College London | Medicine | University | Not in cards | May 2020 | ||||
| Evidence Sufficiency Evaluation Models classify whether the evidence supports an answer across 72 cases. The harness uses paired evidence changes, controls, and strict label matching. | Enas Amin | Faithful to sources | School | Not in cards | Oct 2026 | ||||
| EvoArena PersonaMem-Evo Agents solve questions about changing user preferences. The score is the published accuracy across versioned tasks. | MIT | Learning from the context | Expert | Not in cards | Jun 2026 | ||||
| EvoArena SWE-Chain-Evo Agents solve software tasks that change across versions. The score is the published accuracy across versioned tasks. | MIT | Learning from the context | Expert | Not in cards | Jun 2026 | ||||
| EVVE Models retrieve other videos showing the same real-world event as a query video. | Jérôme Revaud and colleagues | Video understanding | School | Not in cards | Jun 2013 | ||||
| EWMBench Embodied world models generate robot manipulation videos from language. Scored on scene consistency, motion correctness, semantic alignment. | AgiBot | Video world models | Not in cards | May 2025 | |||||
| Exam-et Models answer multiple-choice questions from Estonian high school tests. | TalTechNLP | Knowledge in many languages | School | Not in cards | Sep 2024 | ||||
| EXAMS Models answer high school exam questions in 16 languages across many subjects. Arabic reports use the Arabic subset. | Sofia University and collaborators (Hardalov et al.) | Knowledge in many languages | School | Not in cards | Nov 2020 | ||||
| ExeWorldBench Coding agents build executable code-based 3D worlds that capture how a world works, testing world understanding and its realization in code. | University of Science and Technology of China | Physical prediction | Expert | Not in cards | Oct 2026 | ||||
| ExFunTube Models explain why a moment in a short video is funny. The authors compare explanations with human references and ask viewers to rate them. | Seoul National University and Allen Institute for AI | Video captioning | Everyday | Not in cards | Jul 2023 | ||||
| Expert Forecasts of Real-Effort Experiment Models predict how incentives change the number of alternating key presses workers make in ten minutes. Researchers compare predictions with observed effort across 15 conditions. | Stefano DellaVigna and Devin Pope | Social simulation | University | Not in cards | Jul 2015 | ||||
| ExPUNations Models explain English puns and write new ones from keywords. Human raters check whether a generated pun supports both meanings. | Amazon Alexa AI, University of Southern California and UCLA | Creative writing | Everyday | Not in cards | Oct 2022 | ||||
| ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. | Tsinghua University and collaborators | Simulated robots | Everyday | Not in cards | Nov 2026 | ||||
| EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. | Google (EyePACS images) | Medicine | Expert | Not in cards | Jul 2025 | ||||
| Factorio Play Agents build factories one action at a time in a Factorio simulator. The evaluator measures factory output on held-out layouts. | divagr | Video games | University | Not in cards | Sep 2026 | ||||
| Facts about Singapore Models answer multiple-choice and true-or-false questions about Singapore. An exact match with the answer key earns credit. | AI Verify Foundation | Short factual questions | Everyday | Not in cards | May 2024 | ||||
| FAD Music Listening Test Models estimate acoustic and musical quality for ten-second music excerpts. The test compares predictions with human ratings from ten listeners. | Microsoft Research and Azalea Gui | Audio understanding and voice | Everyday | Not in cards | Nov 2023 | ||||
| FairFace Models classify face images using the dataset’s annotated age, gender, and race categories. The benchmark compares recognition performance across demographic groups. | Kimmo Kärkkäinen and colleagues | General visual questions | Everyday | Not in cards | Aug 2019 | ||||
| FairStress Models choose between candidates as demographic cues and persuasive statements change. The benchmark measures how often models change a correct decision. | Allen Institute for AI | Bias, fairness and politics | Everyday | Not in cards | Oct 2026 | ||||
| FaithDial Models produce conversational answers grounded in supplied knowledge. | Nouha Dziri and colleagues | Faithful to sources | School | Not in cards | Apr 2022 | ||||
| Fake-or-Real Models distinguish synthetic speech from real recordings. NVIDIA evaluates this task within AIR-Bench. | Ricardo Reimao and Vassilios Tzerpos | Audio understanding and voice | Everyday | Not in cards | |||||
| Fake.Br Corpus Models classify Brazilian Portuguese news articles as true or false. The corpus pairs 3,600 false articles with reports on the same topics and provides length-matched versions. | Interinstitutional Center for Computational Linguistics | Faithful to sources | University | Not in cards | Aug 2018 | ||||
| FakeRecogna Models classify Brazilian news articles as true or false. The authors collected 5,951 items per class from different sources, so outlet and writing style can influence predictions. | São Paulo State University | Faithful to sources | University | Not in cards | Jun 2021 | ||||
| FakeTrueBR Models classify Brazilian Portuguese news articles as true or false. The authors paired false stories from Boatos.org with related reporting from G1 and Folha de S.Paulo. | Federal University of Santa Catarina and London School of Economics and Political Science | Faithful to sources | University | Not in cards | Mar 2023 | ||||
| Fashion IQ Models retrieve a garment using a reference photograph and text describing how the target garment differs. | Hui Wu and colleagues | General visual questions | Everyday | Not in cards | May 2019 | ||||
| FastAPI Competency Models implement FastAPI applications from written specifications. The grader issues HTTP requests and checks responses across 34 tasks. | toolshed | Building whole projects | University | Not in cards | Jul 2026 | ||||
| FDA and CDC VAERS clinical and temporal annotations Models locate clinical features and time expressions in vaccine safety reports. Annotators also linked clinical features to their timing across 1,000 reports. | US Food and Drug Administration and Centers for Disease Control and Prevention | Medicine | University | Not in cards | Jun 2018 | ||||
| FDA Information Extraction (Based) Models extract requested field values from chunks of FDA documents. The scorer checks whether the expected value appears in the answer. | Hazy Research (Based evaluation) | Faithful to sources | University | Not in cards | Feb 2024 | ||||
| FEABench Models write Java commands that build and run engineering simulations in COMSOL. The evaluator checks execution, model features, and numerical results where reference values exist. | Google Research and Harvard University | Scientific code | Expert | Not in cards | |||||
| FeedbackQA Models answer questions using a document collection and learn from written feedback about earlier answers. | Zichao Li and colleagues | Questions that need search | School | Not in cards | Apr 2022 | ||||
| Feishu Office v1 Agents handle 140 Chinese workplace requests in a simulated Feishu workspace. The grader checks the resulting calendars, documents, tables, and approvals. | emerge | Documents and spreadsheets | University | Not in cards | Jul 2026 | ||||
| FenBench Models identify chess pieces, locate them, count them or describe a board position in Forsyth-Edwards Notation. The test contains 200 examples. | puccibets | Spatial reasoning and counting | Everyday | Not in cards | Oct 2025 | ||||
| FeruzaSpeech Models transcribe one woman reading Uzbek. The reference text is available in Latin and Cyrillic scripts. | Anna Povey and Katherine Povey | Speech recognition and translation | Everyday | Not in cards | Mar 2024 | ||||
| fev-bench Models forecast time series across 100 tasks, including tasks with additional variables available to the forecaster. | AutoGluon team | Real-world events | University | Not in cards | Sep 2025 | ||||
| FEVER Systems check 185,445 claims against Wikipedia and label each Supported, Refuted or NotEnoughInfo. Annotators wrote the claims by altering Wikipedia sentences. The full FEVER score also requires the right evidence sentences. | University of Sheffield and Amazon (Thorne, Vlachos, Christodoulopoulos, Mittal) | Faithful to sources | Everyday | Not in cards | Mar 2018 | ||||
| FilBench Models perform 12 tasks in Philippine languages. The tasks test cultural knowledge, language processing, reading comprehension and generation. | FilBench team and Hugging Face | Knowledge in many languages | School | Not in cards | Aug 2025 | ||||
| Fin-Eva Models answer 13,106 Chinese multiple-choice questions about financial knowledge, calculations, analysis, and compliance. The evaluator checks the chosen option against the answer key. | Ant Group and Shanghai University of Finance and Economics | Finance | Expert | Not in cards | Sep 2023 | ||||
| Financial Statement OCR Models extract financial figures from profit and loss statements. Ramp counts values within one percent of a risk operator’s transcription. | Ramp Labs | Finance | University | Not in cards | Mar 2026 | ||||
| Financial Statement Review (fs-review) Models review eight sets of financial statements and flag inconsistencies in cash, retained earnings, or the balance sheet. Some sets contain no errors. | martian56 | Finance | University | Not in cards | Jul 2026 | ||||
| FIND Models find inconsistent evidence spans in long technical and financial documents and describe how those spans conflict. | Kensho Technologies, S&P Global | Reading comprehension | Expert | Not in cards | May 2026 | ||||
| FinDVer Models verify claims using text and tables in financial reports. The authors score the final true or false judgment across three kinds of reasoning. | Yale NLP FinDVer team | Finance | University | Not in cards | Sep 2024 | ||||
| FinEvaReranking Models rank Chinese financial news passages for an investor question. Each question has one matching passage and nine or ten distractors. | The Hong Kong University of Science and Technology | Finance | University | Not in cards | Jul 2024 | ||||
| FinFactReranking Models rank fact checking articles for a claim. Each test row includes one designated article and nine or ten distractors. | The Hong Kong University of Science and Technology | Finance | Everyday | Not in cards | Jul 2024 | ||||
| FinMTEB Models represent English and Chinese financial texts as vectors. The suite tests seven abilities, including retrieving definitions, grouping complaints, and comparing financial statements. | The Hong Kong University of Science and Technology | Finance | University | Not in cards | Sep 2024 | ||||
| FinMTEB FINAL Models compare passages from financial reports. The score measures how well their similarity rankings distinguish revised passages from mismatched passages. | The Hong Kong University of Science and Technology | Finance | University | Not in cards | Jul 2024 | ||||
| FinQA Models answer numerical questions about company earnings reports that mix tables and text. Financial experts wrote 8,281 question and answer pairs, each with a gold reasoning program. Answers are checked by execution accuracy and program accuracy. | UC Santa Barbara and J.P. Morgan AI Research | Finance | University | Not in cards | Sep 2021 | ||||
| FinSBD 2021 Models locate document structures in English and French financial PDFs. The third edition adds boundaries for tables, figures, headers, and footers. | Fortia Financial Solutions | Finance | School | Not in cards | Dec 2020 | ||||
| FinSearchComp Greater China Agents search for financial information about Greater China assets across three task types. The score is the published average accuracy for the Greater China subset. | ByteDance Seed | Finance | Expert | Not in cards | Sep 2025 | ||||
| FinSTS Models compare statements from different annual reports by the same company and score whether their financial meaning changed. People labeled 370 test pairs. | The Hong Kong University of Science and Technology | Finance | University | Not in cards | Mar 2024 | ||||
| FiQA2018Reranking Models rank eleven candidate answers to personal finance questions. Each test row has one designated answer. | The Hong Kong University of Science and Technology | Finance | Everyday | Not in cards | Jul 2024 | ||||
| Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. | Fireworks AI | Composite indices | Expert | Not in cards | Sep 2026 | ||||
| FiVE-Bench Models edit objects in videos from written instructions. A vision language model checks whether each edit succeeded, and other metrics measure changes to the background. | Harvard University, Hong Kong Polytechnic University, and City University of Hong Kong | Video generation | University | Not in cards | Mar 2025 | ||||
| FlashDexRetarget 50-motion benchmark Retarget 50 human hand-object motions (HOT3D, TACO, OakInk2) to XHand and Sharpa hands in Isaac Sim. Scored by object pose-error success thresholds. | KAIST AI + Holiday Robotics | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| Flickr30k Models describe photographs of people, objects, and activities. Meta compares model captions with the human reference captions. | Peter Young and colleagues | Detection and captioning | Everyday | Not in cards | |||||
| FLOW-BENCH Models translate natural-language business requirements into executable enterprise workflows. The evaluator checks constrained workflow syntax and behavior. | IBM Research | Apps and MCP servers | Expert | Not in cards | Mar 2025 | ||||
| FoNE Models identify named entities in Faroese news articles. The authors manually verified the automatically assigned labels. | Vésteinn Snæbjarnarson and colleagues | Reading in many languages | School | Not in cards | May 2023 | ||||
| Food-101 Models identify the dish in a food photograph from 101 categories. | ETH Zurich | General visual questions | Everyday | Not in cards | Sep 2014 | ||||
| FORB Models retrieve other images of the same flat object despite changes in viewpoint or lighting. | Pengxiang Wu and colleagues | General visual questions | School | Not in cards | Sep 2023 | ||||
| Formal Conjectures Models prove or disprove mathematical conjectures formalized in Lean. The collection includes open research problems and records machine-checked solutions. | Google DeepMind | Formal proofs | Research frontier | Not in cards | |||||
| Formal-IMO Models prove formal statements of International Mathematical Olympiad problems in Lean. The repository supports partial credit when a problem has multiple parts. | Google DeepMind | Formal proofs | Competition | Not in cards | May 2025 | ||||
| Formally Verified C Models write C functions that satisfy fixed ACSL contracts. Frama-C checks the proofs, and the Core-v1 dataset reserves 16 of 64 tasks for testing. | stanley-ngugi | Function-level problems | Expert | Not in cards | Sep 2026 | ||||
| FormExtract Bench Models extract named fields from form text into JSON. The grader compares each field with its reference and checks that the object matches the schema. | Aishwarya Jagadish | Documents, charts and text in images | School | Not in cards | Oct 2026 | ||||
| Fox Document Benchmark Models read English and Chinese documents and perform tasks including OCR, translation, and question answering. The hub port covers nine subtasks. | Fox authors | Documents, charts and text in images | University | Not in cards | May 2024 | ||||
| FPL 2026 Agentic FPGA Backend Optimization Competition Teams improve the maximum clock frequency of routed FPGA designs while preserving behavior. The organizers rank teams across seven designs after accounting for runtime and model cost. | FPL 2026 competition organizers | Electronics | Expert | Not in cards | Feb 2026 | ||||
| FQuAD 2.0 Models answer questions about French passages and identify questions that the passage cannot answer. | Quentin Heinrich and colleagues | Reading in many languages | School | Not in cards | Sep 2021 | ||||
| FQuAD 1.1 Replaced in Sep 2021 | Martin d’Hoffschmidt and colleagues | Not in cards | May 2020 | ||||||
| FQuAD 1.0 Replaced in May 2020 | Martin d’Hoffschmidt and colleagues | Not in cards | Feb 2020 | ||||||
| FreiHAND Models recover a hand's joint positions and surface from a color image. The evaluation compares predictions with reference hand geometry. | University of Freiburg | Spatial reasoning and counting | School | Not in cards | Sep 2019 | ||||
| French Insurance Job Offer Models mark and classify soft skills in French job ads from Canadian insurers. A domain expert annotated 47 ads. | Université Laval and four Canadian insurers | Other professions | School | Not in cards | Apr 2022 | ||||
| FrenchBench (CroissantLLM) Models answer questions and perform language understanding and generation tasks in French. The suite combines existing tests with manually annotated material. | Manuel Faysse and colleagues | Reading in many languages | School | Not in cards | Feb 2024 | ||||
| FrontierMath: Open Problems Models attack unsolved research math problems proposed by working mathematicians. Each problem has a program that checks a proposed solution. Epoch tracks which problems AI has solved. | Epoch AI | Research mathematics | Research frontier | Not in cards | Jan 2026 | ||||
| Fruit Box Models select rectangles whose entries sum to ten on a ten-by-seventeen grid. The evaluator compares the final score with an expert trajectory from the training split. | djdumpling | Puzzles and chess | School | Not in cards | Oct 2025 | ||||
| FSD50K Models identify the sound events present in a Freesound recording. | Eduardo Fonseca and colleagues | Audio understanding and voice | Everyday | Not in cards | Oct 2020 | ||||
| FSDKaggle2018 Models assign a sound event label to a Freesound recording. | Eduardo Fonseca and colleagues | Audio understanding and voice | Everyday | Not in cards | Mar 2018 | ||||
| FSDKaggle2019 Models identify all relevant sound events in an audio clip using training labels of varying reliability. | Eduardo Fonseca and colleagues | Audio understanding and voice | School | Not in cards | Jun 2019 | ||||
| Fundus-MMBench Models identify retinal structures, classify diseases, and grade their severity in fundus photographs. Each of the 31 tasks contains 20 questions. | Xinyao Liu and Diping Song | Medicine | Expert | Not in cards | Jul 2025 | ||||
| FunGraph3D Models identify objects and their controls in indoor scans. They predict which part operates which object, such as a switch that controls a light. | OpenFunGraph team | Spatial reasoning and counting | Everyday | Not in cards | Mar 2025 | ||||
| FutureEval (Metaculus) Models forecast open Metaculus questions with a fixed prompt and are scored as the questions resolve. Seasonal bot tournaments and Pro Forecaster baselines run alongside. | Metaculus | Real-world events | Expert | Not in cards | Feb 2026 | ||||
| G2 Voice AI Agent Platform Evaluations Voice platforms run the same agent on simulated business calls. Bluejay checks required tool arguments and the database after hangup, then measures transcription and latency. | G2 and Bluejay | Customer service | University | Not in cards | Oct 2026 | ||||
| Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. | phi-monster (Galahad authors) | Simulated robots | Everyday | Not in cards | Jul 2026 | ||||
| GalCoLA Models judge the grammatical acceptability of Galician sentences. | Proxecto Nós | Reading in many languages | School | Not in cards | Jun 2024 | ||||
| GameForgeBench Agents repair game projects in engines such as Godot. The verifier runs the project and checks behavior such as camera movement and collisions. | GamePhanesStudio | Fixing real repositories | University | Not in cards | Sep 2026 | ||||
| GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. | Bosch + NVIDIA + UC Berkeley | Simulated robots | Everyday | Not in cards | Jul 2026 | ||||
| Gauntlet Agents develop game controllers through practice. The authors freeze each controller and test it on unseen games without further model calls. | Haonan Huang and Joey Xiao | Video games | University | Not in cards | Jul 2026 | ||||
| GAUSS Models solve mathematics problems covering 12 skills, including proof checking and mathematical communication. Contributors provide a reference solution and a separate scoring rubric for each problem. | GAUSS contributors | Research mathematics | Expert | Not in cards | Sep 2025 | ||||
| gBench Models run academic evaluations, Golden Set smoke tests, and an agentic quality suite through an OpenAI-compatible endpoint. The tool also records serving performance. | Google DeepMind | Machine learning engineering | University | Not in cards | Jan 2025 | ||||
| GD-VCR Models answer questions about what people are doing and why. The test pairs 328 images, mainly from film and television, with 886 questions across four regions. | University of California, Los Angeles | General visual questions | Everyday | Not in cards | Sep 2021 | ||||
| GDPWorks Agents complete 83 professional tasks across 36 occupations, using source files to produce work such as documents, spreadsheets, and code. The Joules-to-GDP framework compares outputs with expert deliverables through a blinded model judge. Task success measures the share of wins and ties against the expert. | Snorkel AI and Hazy Research (Scaling Intelligence) | Tasks from real jobs | Expert | Not in cards | Nov 2026 | ||||
| Gecko People compare text to image outputs for prompt alignment. The benchmark studies model ordering, pairwise instance scoring, and pointwise instance scoring. | Google DeepMind | Image generation and editing | Everyday | Not in cards | Apr 2024 | ||||
| Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. | Google DeepMind | Deception and alignment | Not in cards | ||||||
| Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. | Google DeepMind | Refusing harmful requests | Not in cards | ||||||
| Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. | Google DeepMind | Jailbreaks and prompt injection | Not in cards | ||||||
| Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. | Google DeepMind | Real robots | Everyday | Not in cards | Mar 2025 | ||||
| Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. | Google DeepMind | Real robots | Everyday | Not in cards | Jul 2026 | ||||
| Gender Resolution in Speaker-Listener Dialogue Roles Models translate English dialogue into Czech, Icelandic or Spanish. The authors score adjective agreement when context identifies gender and measure gender choices when it leaves gender open. | National Research Council Canada | Translation | Everyday | Not in cards | Nov 2024 | ||||
| GENDER1PERSON Models translate 1,000 English product reviews into Russian or Serbian. The test measures the difference between first-person masculine and feminine choices as a percentage of all reviews. | IU University and University of Hildesheim | Translation | Everyday | Not in cards | Jun 2025 | ||||
| General Agent Benchmark (Moonshot) Agents complete multi-step production workflows. Experts compare their work with human expert performance. | Moonshot AI | Tasks from real jobs | Expert | Not in cards | Jan 2026 | ||||
| Generating Confusion: Voting with a Disability Models answer questions about accessible voting in the United States. Reviewers measure how often answers to standard voter questions contain incorrect information. | Center for Democracy & Technology | Long-form factuality | Everyday | Not in cards | Sep 2024 | ||||
| GenEval 2 Models generate images from compositional prompts. A visual question answering model checks whether each requested object, attribute, count, and relation appears. | Meta FAIR, University of Washington, UCLA, and Allen Institute for AI | Image generation and editing | Everyday | Not in cards | Dec 2025 | ||||
| GenExam Image generation systems are evaluated on subject-focused educational tasks with expert ratings. | Google DeepMind | Image generation and editing | University | Not in cards | |||||
| GenomeGym-BRCA1 Models estimate whether a documented BRCA1 variant causes loss of function. The grader compares each probability with a hidden functional label. | alfaxad | Genomics and life sciences | Expert | Not in cards | May 2026 | ||||
| GenScale Models generate or edit images containing objects with known relative sizes. A judge calibrated against human ratings scores the relative scale of object pairs. | Boston University and collaborators | Image generation and editing | Everyday | Not in cards | Sep 2026 | ||||
| GEO-Bench-2 Models interpret Earth imagery across 19 tasks, including field boundaries, forest biomass and tree species. The authors normalize task scores to compare model capabilities. | AI Alliance GEO-Bench-2 team | Spatial reasoning and counting | University | Not in cards | Nov 2025 | ||||
| GeoDE Models recognize objects in images collected across different world regions. The evaluation compares accuracy across regions and object classes. | GeoDE team | General visual questions | Everyday | Not in cards | Jan 2023 | ||||
| GeoExpertQueries Agents choose geospatial data and carry out an analysis. A model compares each execution trace with a reference workflow agreed on by three GIS specialists. | Google Research | Apps and MCP servers | Expert | Not in cards | Oct 2026 | ||||
| GeoKTools Agents combine up to four kinds of geospatial tools to answer quantitative questions. Exact matches earn full credit, and a model grades other answers. | Google Research | Apps and MCP servers | University | Not in cards | Oct 2026 | ||||
| GeoMLAMA Models complete sentences about daily life in five countries. The test checks whether models rank locally correct answers above other choices in five languages. | University of California, Los Angeles | Knowledge in many languages | Everyday | Not in cards | May 2022 | ||||
| GeoSound Models match environmental recordings to satellite images of their locations and retrieve recordings from images. The test holds out geographic areas and contains 9,931 examples. | Subash Khanal and colleagues | Audio understanding and voice | Everyday | Not in cards | Aug 2024 | ||||
| GerDaLIR Models find the German court decisions cited by a passage after the authors mask its references. The original test split has 12,298 queries. | Marco Wrzalik and Dirk Krechel | Law | University | Not in cards | Oct 2021 | ||||
| German Argument Mining Models label sentences from Bavarian court judgments as conclusions, definitions, applications of law to facts, or other text. The authors report F1 and accuracy. | Technical University of Munich and University of Passau | Law | University | Not in cards | Jul 2020 | ||||
| German federal election chatbot audit Chatbots answer German election questions in English and German. Researchers count correct electoral answers out of 30 per chatbot and rate political bias separately. | Democracy Reporting International | Long-form factuality | Everyday | Not in cards | Feb 2025 | ||||
| German Traffic Sign Recognition Benchmark Models identify the class of a traffic sign in a cropped road image. | Johannes Stallkamp and colleagues | Detection and captioning | Everyday | Not in cards | Dec 2010 | ||||
| GermEval 2014 Named Entity Recognition Models find and classify names in German news and Wikipedia text, including nested and derived entities. | Darina Benikova and colleagues | Reading in many languages | School | Not in cards | May 2014 | ||||
| GIE-Bench Image-editing models follow text instructions while preserving unedited regions. GPT-4o question answers and masked image metrics score functional correctness and preservation. | Apple | General visual questions | University | Not in cards | May 2025 | ||||
| GIFT-Eval Models forecast future values from historical time series across multiple domains and forecast horizons. | Salesforce AI Research | Real-world events | University | Not in cards | Oct 2024 | ||||
| GigaAM Multilingual Internal In-the-Wild Speech Tests Models transcribe crowdsourced recordings in Russian, Kazakh, Kyrgyz and Uzbek. The evaluators report word error rate by language. | Sber | Speech recognition and translation | Everyday | Not in cards | Jul 2026 | ||||
| GigaSpeech Models transcribe English speech from audiobooks, podcasts, and online videos. The test compares transcripts using word error rate. | GigaSpeech team | Speech recognition and translation | Everyday | Not in cards | Jun 2021 | ||||
| GLES 2017 vote choice Models use a German survey respondent’s profile to estimate their reported vote in the 2017 federal election. Researchers compare individual choices and party vote shares. | Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz | Social simulation | Everyday | Not in cards | Dec 2023 | ||||
| GlobalMGSM Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number. | Cohere Labs | Reasoning in many languages | School | Not in cards | Mar 2026 | ||||
| GLUE Models predict labels or similarity scores across nine English language-understanding tasks. The overall score averages task scores. Tasks with several metrics first average those metrics. | New York University, University of Washington, and DeepMind | Language reasoning | Everyday | Not in cards | Apr 2018 | ||||
| Gold Commodity News Models classify news headlines about gold. Labels distinguish price direction, timing, and comparisons with other assets. | Ankur Sinha and colleagues | Finance | School | Not in cards | Sep 2020 | ||||
| Goliath human mesh recovery Models reconstruct a person's body from images of studio performances. Meta measures joint and surface errors against captured three-dimensional motion. | Meta | Spatial reasoning and counting | School | Not in cards | |||||
| Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. | Google DeepMind | Dangerous capabilities | Not in cards | ||||||
| Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. | Google DeepMind (with Wiz) | Finding and exploiting real bugs | Expert | Not in cards | |||||
| Google Landmarks Dataset v2 Models recognize landmarks or retrieve photographs of the same landmark. The benchmark uses images from around the world. | General visual questions | Everyday | Not in cards | Apr 2020 | |||||
| Google Scanned Objects Models synthesize an object from a new viewpoint. Researchers compare the rendered result with reference views of scanned household items. | Google Research | Spatial reasoning and counting | School | Not in cards | Apr 2022 | ||||
| Google Universal Image Embedding Challenge Models retrieve images of the same item across several visual domains using a single image representation. The challenge compares results against labeled matches. | General visual questions | Everyday | Not in cards | Aug 2022 | |||||
| Gorgias AI Agent Competitive Benchmark Agents recommend products and answer support questions on live Shopify stores. Gorgias combines answer quality, response time, and the share of conversations handled without human help. | Gorgias | Running a business | Everyday | Not in cards | Jul 2026 | ||||
| GovReport Models summarize long US government reports from the Government Accountability Office and the Congressional Research Service. The reports are much longer than earlier summarization sets, and each has an expert-written summary. Scores are ROUGE overlap with the reference summary. | University of Michigan and UIUC | Reasoning over long documents | University | Not in cards | Apr 2021 | ||||
| GPT-3 Arithmetic Models calculate answers to ten types of arithmetic problems. The tasks vary the operation and the number of digits. | OpenAI | Word problems | School | Not in cards | May 2020 | ||||
| GPT-3 Word Scrambling Models recover English words after their letters are rearranged or extra characters are inserted. The test measures exact recovery across several transformations. | OpenAI | Language reasoning | School | Not in cards | May 2020 | ||||
| Grade Like a Human OS Dataset Models grade undergraduate operating systems answers. The authors measure absolute error against the mean grade from three teaching assistants per answer. | Wenjing Xie and colleagues at City University of Hong Kong and Mohamed bin Zayed University of Artificial Intelligence | Other professions | University | Not in cards | May 2024 | ||||
| GRAFT Models predict plant traits from gene expression in Arabidopsis. The authors compare regression and graph models against measurements from the same plants. | Manuel Serna-Aguilera and colleagues | Genomics and life sciences | University | Not in cards | Jun 2026 | ||||
| GrailQA Models answer questions by composing queries over Freebase. The evaluator measures semantic logical-form exact match and answer-set F1 across ordinary, compositional, and unseen-schema generalization settings. | Ohio State University, U.S. Army Research Laboratory, Stanford University and others | Language reasoning | Everyday | Not in cards | Nov 2020 | ||||
| Grammarly Corpus of Discourse Coherence Models rate the coherence of emails, reviews, and answers to online questions. Each of four collections has 200 test texts with expert ratings. | Grammarly and the University of Illinois at Urbana-Champaign | Reading comprehension | School | Not in cards | Jul 2018 | ||||
| GraspBench Robots propose grasps and try to lift objects in MuJoCo. Each attempt must pass motion planning and execution on the chosen robot. | Abdul Basit and colleagues | Simulated robots | Everyday | Not in cards | Nov 2026 | ||||
| Gray Swan Shade (adaptive attacker) Gray Swan's adaptive red-teaming agent trained on test scenarios to attack coding, computer-use and browser agents with prompt injections. | Gray Swan | Jailbreaks and prompt injection | Not in cards | ||||||
| Green Job Description Entity Recognition Models mark spans for skills, qualifications, experience, occupations, and domains in UK job ads. Human annotations supply the answer key. | University of Sheffield | Other professions | School | Not in cards | Jun 2022 | ||||
| Greptile AI Code Review Benchmark AI code review tools inspect 50 pull requests from five open-source repositories and report bug detection, precision, review quality and language coverage. | Greptile | Fixing real repositories | Expert | Not in cards | Jan 2025 | ||||
| Gretel Synthetic PII Finance Models identify and label personal information in synthetic financial documents across seven languages. Independent evaluations measure how well the predicted spans match the labeled spans. | Gretel | Finance | Everyday | Not in cards | Jun 2024 | ||||
| Grooming the Machine Chatbots respond to three false claims under four user personas. The researcher counts a repeated false claim or a refusal as a failure. | Arianna Stech, European University Institute | Long-form factuality | Everyday | Not in cards | Apr 2026 | ||||
| GroupAssistBench Assistants use memories from group conversations to answer questions and complete real-world group assistance tasks. Model judges score grounded helpfulness. | Tencent | Learning from the context | Expert | Not in cards | Nov 2026 | ||||
| GS-QA2 Models answer questions about terrain and map features using elevation grids and vector data. The grader checks answers against PostGIS queries with tolerances for each measurement. | University of California, Riverside | Apps and MCP servers | University | Not in cards | May 2026 | ||||
| GTFS Retrieval Models query a Chicago bus schedule from four routes. They answer directly or generate Python code whose output is compared with a reference answer. | University of Illinois Urbana-Champaign | Expert-written questions | University | Not in cards | Jul 2023 | ||||
| GTFS Semantics Models answer multiple-choice questions about the General Transit Feed Specification. Questions cover transit data files, field definitions, and relationships between tables. | University of Illinois Urbana-Champaign | Expert-written questions | University | Not in cards | Jul 2023 | ||||
| GTZAN Models identify the music genre in short recordings. Qwen reports genre classification accuracy across music understanding systems. | George Tzanetakis and Perry Cook | Audio understanding and voice | Everyday | Not in cards | Jul 2002 | ||||
| GutBrainIE 2026 Models find entities in 80 gut and brain research abstracts, link them to biomedical concepts, and identify relationships between mentions and between concepts. | University of Padua and collaborators | Medicine | University | Not in cards | Feb 2026 | ||||
| GutBrainIE 2025 Replaced in Feb 2026 | Not in cards | Feb 2025 | |||||||
| HAGRID Models answer information-seeking questions and attribute their claims to supporting passages. | Ehsan Kamalloo and colleagues | Faithful to sources | University | Not in cards | Jul 2023 | ||||
| Hallmarks of Cancer Models label biomedical text with the cancer hallmarks it describes. The hub port assigns zero or more of ten labels to individual sentences. | Simon Baker and colleagues | Genomics and life sciences | University | Not in cards | Oct 2015 | ||||
| HalloMTBench Models translate English text into one of 11 languages. Judges count translations that add unsupported material, use the wrong language, leave text untranslated or repeat material. | Alibaba International Digital Commerce and Tianjin University | Translation | University | Not in cards | Oct 2025 | ||||
| HalluLens Language models answer dynamically generated prompts that test factual and contextual hallucination. The evaluator builds fresh examples to reduce test leakage. | Meta FAIR and Superintelligence Labs | Hallucination rates | Everyday | Not in cards | Apr 2025 | ||||
| HalluScoring 2026 Models detect factual errors in Arabic answers. Separate tasks test unseen questions, unseen answer generators and the ability to select a correct alternative. | HalluScoring organizers | Hallucination rates | University | Not in cards | May 2026 | ||||
| Hallym University Women's Health Nursing Exam Models answer 50 Korean questions from a fourth-year nursing course. The authors compare accuracy and estimated ability with results from 111 students. | Hallym University | Medicine | University | Not in cards | Sep 2025 | ||||
| HaluEvalAudio 1000 Audio language models answer spoken questions against supplied knowledge and reference answers. | Kyutai | Hallucination rates | University | Not in cards | Mar 2026 | ||||
| HaluMem Agent memory systems are tested for hallucinations at each operation, memory extraction, updating and question answering, rather than only end to end. | MemTensor | Hallucination rates | Everyday | Not in cards | Nov 2025 | ||||
| Hands Visible Models inspect 12 frames per clip and determine whether a human hand or robot gripper appears. The 45-clip evaluation uses provisional machine-generated labels. | referee-lab | Video understanding | Everyday | Not in cards | Aug 2026 | ||||
| Hangman Bench Agents guess letters to identify hidden words in Hangman. The harness tracks wins across difficulty levels. | MattFisher | Puzzles and chess | Everyday | Not in cards | Jun 2025 | ||||
| HANS An entailment test built to catch three shallow shortcuts: lexical overlap, subsequence and constituent. Every premise and hypothesis pair shares words, but only some pairs are entailments. A model that relies on word overlap fails the non-entailment cases. | Johns Hopkins University and Brown University (McCoy, Pavlick, Linzen) | Language reasoning | Everyday | Not in cards | Feb 2019 | ||||
| HappyWorld-Bench World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them. | Alibaba | Interactive world models | Everyday | Not in cards | Sep 2026 | ||||
| HARMOGEN-R Models grade programming students’ responses using human and model-written rubrics. The dataset includes 770 responses from a data structures and algorithms course. | University of Madeira | Other professions | University | Not in cards | Oct 2025 | ||||
| Harmony4D Models recover the body shapes and poses of people interacting at close range. The benchmark includes sports where people occlude each other. | Rawal Khirodkar and colleagues | Spatial reasoning and counting | School | Not in cards | Oct 2024 | ||||
| Harness Leaderboard Coding-agent harnesses such as Claude Code, Codex and OpenCode run the same open model at the same reasoning effort on 198 tasks from Terminal-Bench 2.1 and DeepSWE v1.1. Scored as the resolve rate averaged across the two benchmarks over three attempts, with any access to the answer counted as a fail. | Joel Niklaus (Hugging Face) | Terminal tasks | Expert | Not in cards | Nov 2026 | ||||
| harness-bench-fast Agents operate on files and code using tools and memory. Harbor exports 231 tasks from version 0.3.0. Later public leaderboards report Light and Medium suites separately. | AI Forever | Terminal tasks | University | Not in cards | May 2026 | ||||
| Hateful Memes Challenge Models decide whether a meme is hateful by interpreting its image and caption together. Paired examples change one modality while preserving the other. | Douwe Kiela and colleagues | Bias, fairness and politics | Everyday | Not in cards | May 2020 | ||||
| HC3Reranking Models rank answers to personal finance questions. Each test question has one relevant human answer and one relevant ChatGPT answer. | The Hong Kong University of Science and Technology | Finance | Everyday | Not in cards | Jul 2024 | ||||
| HCAST Agents attempt 189 machine learning, cybersecurity, software engineering and reasoning tasks. 563 human baselines time each task, from one minute to over 8 hours, so success can be read against human task length. | METR | Task length | Expert | Not in cards | Mar 2025 | ||||
| HDLBits Models write Verilog modules for digital logic exercises. The online judge compiles each submission and compares its simulated outputs with a reference design. | Henry Wong | Electronics | University | Not in cards | |||||
| HEAD-QA Models answer questions from Spanish healthcare specialization exams in six disciplines. The hub port uses the English training and validation splits. | David Vilares and Carlos Gómez-Rodríguez | Medicine | Expert | Not in cards | Jan 2019 | ||||
| HealthAdvice Models classify a sentence from a medical research paper as strong advice, weak advice, or no advice. | Syracuse University and Jun Wang | Medicine | School | Not in cards | Sep 2021 | ||||
| Hebbia Financial Services Benchmark (2025) Models extract facts, summarize documents and reason about financial information. Hebbia evaluates responses to more than 600 questions. | Hebbia | Finance | Expert | Not in cards | Sep 2025 | ||||
| Hebrew Diacritization Test Corpora Models restore vowel and consonant marks in Hebrew prose, rabbinic commentary and liturgical poetry. The reference also identifies spelling letters that vocalized text should omit. | Dicta and Bar-Ilan University | Reading in many languages | University | Not in cards | Jan 2020 | ||||
| Hedge Gym Models adjust an options hedge each day over a 21-day price path. The evaluator measures replication error and transaction costs on synthetic or historical markets. | Sablier | Finance | Expert | Not in cards | Oct 2026 | ||||
| HEIM Text-to-image models are scored on 12 aspects, such as alignment, quality, aesthetics, originality, reasoning, knowledge, bias and toxicity, with human raters and automatic metrics. | Stanford CRFM | Image generation and editing | Everyday | Not in cards | Nov 2023 | ||||
| HexMachina Models write Catan players in code and compete in the board game. | HexMachina authors | Multi-agent games | University | Not in cards | Jun 2025 | ||||
| Hi4D Models reconstruct the bodies of people interacting at close range. The SAM 3D Body evaluation compares joint and mesh errors with motion capture references. | Hi4D team | Spatial reasoning and counting | School | Not in cards | Mar 2023 | ||||
| Hidden Voices Audio Difficulty Models rank piano works by Black women composers from recordings. The reference ordering comes from the Hidden Voices project’s seven difficulty grades. | Universitat Pompeu Fabra, Sogang University, and University of Alicante | Audio understanding and voice | University | Not in cards | Mar 2024 | ||||
| HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. | Zhongyu Xia and colleagues | Simulated robots | Everyday | Not in cards | May 2026 | ||||
| HierText Models locate and recognize text arranged in lines and paragraphs in real scenes. Google also uses the annotations to compare text and image retrieval. | Google Research | Documents, charts and text in images | School | Not in cards | Mar 2022 | ||||
| HIPE-OCRepair 2026 Models correct errors in machine transcriptions of historical newspapers and printed works in English, French, and German. They see the transcriptions without the page images. | HIPE-OCRepair organizers | Reading in many languages | School | Not in cards | Nov 2025 | ||||
| Historical Systems Bugs Agents repair two historical C bugs in Linux and Unikraft. Hidden tests exercise the patched code, including driver behavior under emulated hardware. | morris-biotech | Fixing real repositories | Expert | Not in cards | Nov 2026 | ||||
| HLE Bio/Chem Gold Models answer biology and chemistry questions from Humanity’s Last Exam whose answers reviewers supported with published research. | FutureHouse | Science questions | Expert | Not in cards | Jul 2025 | ||||
| HM3D-MFMON Agents navigate across floors and find a sequence of objects. | BAAI FlagEval | Simulated robots | Everyday | Not in cards | Aug 2026 | ||||
| HM3DSem-Walks Models use indoor color and depth scans to find objects described by room and floor. The original retrieval test covers eight scenes. | University of Freiburg and University of Technology Nuremberg | Spatial reasoning and counting | Everyday | Not in cards | Mar 2024 | ||||
| HMDB51 Models identify human actions in short video clips across 51 categories. | Hilde Kuehne and colleagues | Video understanding | Everyday | Not in cards | Nov 2011 | ||||
| HoloAssist Mistake Detection Models watch a person perform a task and classify each annotated action as correct or mistaken. The challenge reports an F-score. | Xin Wang and colleagues | Video understanding | Everyday | Not in cards | Sep 2023 | ||||
| Homayouni and Fontes Flexible Job Shop with Transportation Agents schedule jobs on eligible machines and route two vehicles between operations. The score measures when the last operation finishes. | S. M. Homayouni and D. B. M. M. Fontes | Tasks from real jobs | University | Not in cards | |||||
| Honor of Kings Game Configuration Agents configure game data with the Honor of Kings spreadsheet platform. The team measures configuration accuracy before and after the agent checks and repairs its work. | Tencent | Tasks from real jobs | University | Not in cards | Nov 2026 | ||||
| Hotter and Colder Models predict sentiment and other attributes of Icelandic blog text using crowdsourced judgments. | Steinunn Rut Friðriksdóttir and colleagues | Reading in many languages | Everyday | Not in cards | Feb 2025 | ||||
| HourVideo Multimodal models answer questions about first-person videos that run 20 to 120 minutes. The 500 videos come from Ego4D, and the 12,976 five-way multiple-choice questions cover summarization, perception, visual reasoning and navigation. Score is accuracy. | Stanford University | Long videos | Everyday | Not in cards | Nov 2024 | ||||
| How frontier models miss risk in routine enterprise work Agents handle workplace requests whose risks appear in the prompt or in supporting files. Judges check whether the agent completes an unsafe action. | Mercor | Refusing harmful requests | Expert | Not in cards | Nov 2026 | ||||
| How2QA Models answer multiple-choice questions about instructional videos. The task tests understanding of the actions shown in each clip. | Microsoft Research | Video understanding | Everyday | Not in cards | May 2020 | ||||
| Huayu Awards AI Cockpit Evaluation 2026 In-car assistants interpret requests and operate vehicle features. Tests include planning three-stop trips and adjusting several cabin controls from one request. | J.D. Power China and Tongji University HVR Lab and XAI Lab | General assistants | Everyday | Not in cards | Sep 2026 | ||||
| HubBench Agents make workplace decisions across 13 professional domains using connected software tools and evidence. Version 1.4.0 grades 104 tasks with deterministic state checks. | Blobfish AI | Tasks from real jobs | Expert | Not in cards | Sep 2026 | ||||
| Hugging Face Speech Bench Speech recognition models transcribe recordings. The board collects error rates from model cards. | Hugging Face | Speech recognition and translation | Everyday | Not in cards | Mar 2022 | ||||
| Human ChatGPT Comparison Corpus Models predict whether a person or ChatGPT wrote an answer in English or Chinese. The evaluation includes complete answers and individual sentences. | Hello-SimpleAI and the HC3 authors | Reading in many languages | Everyday | Not in cards | Jan 2023 | ||||
| HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. | UC Berkeley + Yonsei University (Sferrazza et al.) | Simulated robots | Everyday | Not in cards | Mar 2024 | ||||
| Hy-Nomas-Bench Interactive video world models follow text and camera controls across multiple turns and retain spatial memory. | Tencent Hunyuan | Interactive world models | Expert | Not in cards | Oct 2026 | ||||
| HY3D-Bench Models generate textured 3D meshes from text and images, with semantic and geometric quality evaluated across categories. | Tencent Hunyuan | Image generation and editing | Expert | Not in cards | Feb 2026 | ||||
| HydroLLM-Benchmark Models answer hydrology questions drawn from a textbook and research papers. The authors score objective answers by accuracy and other answers by semantic similarity. | University of Iowa Hydroinformatics Lab | Science questions | University | Not in cards | Oct 2024 | ||||
| i2b2 2006 Smoking Challenge Models read a discharge summary and classify the patient's smoking status. The labels include current smoker, past smoker, unspecified smoker, nonsmoker, and unknown. | Informatics for Integrating Biology and the Bedside | Medicine | School | Not in cards | Jul 2006 | ||||
| i2b2 2008 Obesity Challenge Models classify obesity and 15 related conditions from discharge summaries. The challenge scores what the note states explicitly and what a clinician can infer. | Informatics for Integrating Biology and the Bedside | Medicine | Expert | Not in cards | Mar 2008 | ||||
| i2b2 2009 Medication Extraction Challenge Models extract drug names and prescription details from discharge summaries. The organizers scored six fields against community annotations for 251 test summaries. | Informatics for Integrating Biology and the Bedside | Medicine | University | Not in cards | Jun 2009 | ||||
| i2b2/UTHealth 2014 Identifying Risk Factors for Heart Disease over Time Models find heart disease risk factors in notes about patients with diabetes. They mark when each factor was present relative to the note date. | i2b2 and the University of Texas Health Science Center at Houston | Medicine | University | Not in cards | Mar 2014 | ||||
| IARPA Babel speech recognition Models transcribe conversational speech in languages with limited labeled data. Meta measures character error against reference transcripts on the Babel development split. | IARPA | Speech recognition and translation | School | Not in cards | |||||
| ICASSP 2026 Automatic Song Aesthetics Evaluation Challenge Models listen to generated songs and predict musicians’ ratings. One track scores overall musicality, and the other scores five aspects of each performance. | ICASSP ASAE Challenge organizers | Audio understanding and voice | Everyday | Not in cards | Aug 2025 | ||||
| ICDAR 2025 MapText Competition Models detect and read words on historical maps and link them into phrases. This edition adds Taiwanese maps with Chinese labels and revises how links score. | ICDAR MapText organizers | Documents, charts and text in images | School | Not in cards | Dec 2024 | ||||
| ICDAR 2024 MapText Competition Replaced in Dec 2024 | Not in cards | Jan 2024 | |||||||
| ICNALE Edited Essays Models score original English learner essays against an editor’s ratings. The collection pairs each original with an edited version and records scores for five writing traits. | Shin’ichiro Ishikawa and the ICNALE project at Kobe University | Other professions | University | Not in cards | Aug 2016 | ||||
| ICNALE Global Rating Archives Models grade essays and spoken responses in English. The archive provides ratings from 80 people per sample across 140 essays and 140 speech samples. | Shin’ichiro Ishikawa and the ICNALE project at Kobe University | Other professions | University | Not in cards | Oct 2023 | ||||
| IdeaTaste Agents propose research ideas from literature frozen at 2025. Evaluation compares their citations and ideas with 2026 publications and measures diversity. | Carnegie Mellon University | AI research tasks | Research frontier | Not in cards | Nov 2026 | ||||
| IEMOCAP Models identify emotions in recorded dialogues. NVIDIA evaluates this task within AIR-Bench. | University of Southern California | Audio understanding and voice | Everyday | Not in cards | Nov 2008 | ||||
| IIT Madras Hindi ASR Challenge Speech models transcribe Hindi newspaper passages read aloud. The organizers score word errors and rank open and restricted training separately. | Speech Lab, IIT Madras | Speech recognition and translation | Everyday | Not in cards | Jun 2020 | ||||
| IIT Madras Hindi-Tamil-English ASR Challenge Speech models transcribe Hindi, Tamil and Indian English recordings. The organizers report word error rates separately for each language and training condition. | Speech Lab, IIT Madras | Speech recognition and translation | Everyday | Not in cards | May 2021 | ||||
| Image Matching Challenge Models match photographs of landmarks taken from different viewpoints. The evaluation measures the accuracy of camera poses recovered from the matches. | Image Matching Challenge organizers | Spatial reasoning and counting | School | Not in cards | |||||
| ImageCoDe Models select the image that matches a description from a set of visually similar alternatives. | Benno Krojer and colleagues | General visual questions | Everyday | Not in cards | Mar 2022 | ||||
| ImageNet ReaL Models classify ImageNet validation images against revised human annotations. An image can have more than one acceptable label. | Google Research | General visual questions | Everyday | Not in cards | Jun 2020 | ||||
| ImageNet-1K Models assign one of a thousand object labels to each photograph. Meta reports classification accuracy on the validation images. | ImageNet project | General visual questions | Everyday | Not in cards | |||||
| ImageNet-A Models classify real photographs that commonly fool image classifiers. The test uses unmodified images from ImageNet categories. | Dan Hendrycks and colleagues | General visual questions | Everyday | Not in cards | Jul 2019 | ||||
| ImageNet-C Models classify images under common corruptions at different severities. The benchmark measures errors under blur, noise, weather, and digital distortions. | Dan Hendrycks and Thomas Dietterich | General visual questions | Everyday | Not in cards | Mar 2019 | ||||
| ImageNet-R Models classify renditions of objects in drawings, sculptures, and other media. The benchmark measures recognition when appearance differs from ordinary photographs. | Dan Hendrycks and colleagues | General visual questions | Everyday | Not in cards | Jun 2020 | ||||
| ImageNet-Sketch Models classify black-and-white sketches using ImageNet category labels. The benchmark tests recognition when images lack photographic texture and color. | Haohan Wang and colleagues | General visual questions | Everyday | Not in cards | May 2019 | ||||
| ImageNet-UA Models classify images under distortions that differ from those used during training. The updated suite measures accuracy across 18 new distortions. | Center for AI Safety | General visual questions | Everyday | Not in cards | Jul 2023 | ||||
| ImageNetV2 Models classify a newly collected set of photographs using the original ImageNet labels. The test measures how accuracy changes on fresh images. | Benjamin Recht and colleagues | General visual questions | Everyday | Not in cards | Feb 2019 | ||||
| IMDb-PT Models classify the sentiment of Brazilian Portuguese translations of IMDb movie reviews. | Maritaca AI | Reading in many languages | Everyday | Not in cards | |||||
| Inappropriate Image Prompts Models generate images from prompts drawn from real users. The evaluation measures how often the outputs contain inappropriate material. | Patrick Schramowski and colleagues | Refusing harmful requests | Everyday | Not in cards | Nov 2022 | ||||
| INAS hypothesis classification Models read the title and abstract of an invasion biology paper and identify which of ten ecological hypotheses it addresses. A paper can address several hypotheses. | Bielefeld University and collaborators | Lab and data work | University | Not in cards | Nov 2022 | ||||
| INAS hypothesis span prediction Models mark the words in a scientific title or abstract that indicate a particular ecological hypothesis. Experts marked the evidence in 750 invasion biology abstracts. | Bielefeld University and collaborators | Lab and data work | University | Not in cards | Jul 2024 | ||||
| iNaturalist 2021 Models classify nature photographs by species. The 2021 competition uses a new collection of images and species labels. | iNaturalist Competition organizers | General visual questions | University | Not in cards | Mar 2021 | ||||
| iNaturalist 2018 Replaced in Mar 2021 | Not in cards | Feb 2018 | |||||||
| iNaturalist 2017 Replaced in Feb 2018 | Not in cards | Apr 2017 | |||||||
| iNaturalist Sounds Models identify animal species from recordings submitted to iNaturalist. | Mustafa Chasmai and colleagues | Audio understanding and voice | Expert | Not in cards | May 2025 | ||||
| INCLUDE Indian Sign Language Recognition Models identify isolated Indian Sign Language words in video. The authors provide separate test files for the full vocabulary and its smaller selection. | AI4Bharat and IIT Madras | Video understanding | Everyday | Not in cards | Oct 2020 | ||||
| IncompeBench Models retrieve music clips that match detailed natural language requests. | Benjamin Clavié and colleagues | Audio understanding and voice | School | Not in cards | Feb 2026 | ||||
| IndicQA Models extract answers from Wikipedia passages in Indian languages. Native speakers wrote the questions and answers. | AI4Bharat | Reading in many languages | School | Not in cards | Jul 2023 | ||||
| IndicXNLI Models compare a premise and hypothesis in Indian languages. They predict whether the premise entails, contradicts or leaves the hypothesis unresolved. | Divyanshu Aggarwal and colleagues | Reasoning in many languages | School | Not in cards | Apr 2022 | ||||
| IndoNLI Models decide whether a hypothesis follows from an Indonesian passage using annotations from experts and other speakers. | Rahmad Mahendra and colleagues | Reading in many languages | School | Not in cards | Nov 2021 | ||||
| InfraBench Agents deploy, profile, debug, and optimize model workloads on AMD MI35X GPUs. Task-specific verifiers grade 20 engineering assignments. | InfraBench team | Machine learning engineering | Expert | Not in cards | Jul 2026 | ||||
| InnovationEval Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches. | Epoch AI | AI research tasks | Research frontier | Not in cards | Oct 2026 | ||||
| INQUISITIVE Models read a news article up to a highlighted phrase and ask a question a reader might have. Human raters judge validity, relevance, and salience. | University of Texas at Austin | Reading comprehension | Everyday | Not in cards | Sep 2020 | ||||
| Insight Consolidation v1 Agents read customer account documents, infer why the customer acted, and cite supporting records. The evaluator combines a model judgment with citation accuracy. | pavel-substrate | Reasoning over long documents | University | Not in cards | Aug 2026 | ||||
| Inspect India Evals Models answer questions about Indian culture and reason across Indian languages. Other tasks test social bias, digital public infrastructure safeguards and refusal behavior. | Abhishek Kumar Singh and colleagues | Knowledge in many languages | University | Not in cards | Jul 2026 | ||||
| Instinct Next Edit Models predict the next edit from code context and recent changes. The port uses TypeScript, which has 221 held-out examples. | Continue | Function-level problems | University | Not in cards | Sep 2025 | ||||
| InstructIR Models find documents that satisfy a user's search request and its accompanying instructions. | Hanseok Oh and colleagues | Questions that need search | School | Not in cards | Feb 2024 | ||||
| InsureBench Decision Models (Broader Comparison) Models read text evidence and return structured findings about insurance scenarios. The scorer compares 150 findings from 50 packets with the author's labels. | InsureThing | Finance | University | Not in cards | Oct 2026 | ||||
| Internal Finance Modeling (Moonshot) Agents complete financial modeling tasks with linked formulas, financial logic and final workbook quality. Experts rate the work. | Moonshot AI | Finance | Expert | Not in cards | Oct 2026 | ||||
| Internal Financial Research (Moonshot) Agents produce auditable financial research under evidence and cutoff constraints. Experts rate the resulting research. | Moonshot AI | Finance | Expert | Not in cards | Oct 2026 | ||||
| IntervalQA Models estimate numerical answers and give intervals for their uncertainty. The authors measure how often the intervals contain the true answers. | Andy Zou and collaborators | Language reasoning | University | Not in cards | Jun 2022 | ||||
| Invoice Approval (Ali Siala) Agents process 60 supplier invoice work orders under a German company’s approval policy. The evaluator compares the resulting database with a reference state. | Ali Siala | Documents and spreadsheets | University | Not in cards | Nov 2026 | ||||
| IOL-AI 2026 Models infer linguistic rules from Olympiad problems and answer individual questions. The official competition combines exact match and character-level similarity. | IOL-AI organizers and Cohere Labs | Reasoning in many languages | Competition | Not in cards | Jun 2026 | ||||
| IronClad Agents repair security misconfigurations in Terraform repositories. The grader verifies that the repaired configuration satisfies policy and preserves the intended infrastructure. | vinay-k-rajith | Secure code and review | University | Not in cards | Aug 2026 | ||||
| iSAID Models detect and segment individual objects in aerial images. The benchmark measures how well predicted masks match annotated object instances. | Syed Waqas Zamir and colleagues | Detection and captioning | University | Not in cards | May 2019 | ||||
| iSarcasmEval Models identify intended sarcasm in English and Arabic texts labeled by their authors. Other tracks classify ironic speech and distinguish sarcastic text from a literal paraphrase. | Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson, and Walid Magdy | Language reasoning | Everyday | Not in cards | Jul 2022 | ||||
| IsiZulu Second Language Learner Speech Corpus Models detect sounds that learners mispronounce or insert when reading isiZulu. Teachers marked each sound against the intended pronunciation. | Indiana University and University of KwaZulu-Natal | Audio understanding and voice | University | Not in cards | May 2024 | ||||
| IslamicEval 2025 Quotation Correction Models correct inaccurate Quran and Hadith quotations against the supplied sources. The organizers score exact corrections and accept a designated answer for uncorrectable text. | Qatar Computing Research Institute and collaborators | Faithful to sources | School | Not in cards | Jun 2025 | ||||
| IslamicEval 2025 Quotation Span Identification Models mark the character spans of intended Quran and Hadith quotations in Arabic model answers. Organizers score character labels with macro F1. | Qatar Computing Research Institute and collaborators | Faithful to sources | University | Not in cards | Jun 2025 | ||||
| IslamicEval 2025 Quotation Validation Models check whether marked Quran and Hadith quotations agree with their sources. The organizers report accuracy. | Qatar Computing Research Institute and collaborators | Faithful to sources | University | Not in cards | Jun 2025 | ||||
| IslamicEval 2025 Quran and Hadith Passage Retrieval Models rank up to 20 passages from the Quran and Sahih al-Bukhari for an Arabic question. Organizers score mean average precision at ten. | Qatar Computing Research Institute and collaborators | Questions that need search | University | Not in cards | Jun 2025 | ||||
| IslamTrust Models choose answers consistent with scholarly consensus across Sunni schools of thought. The makers provide 406 questions in both Arabic and English. | Abderraouf Lahmar and collaborators | Knowledge in many languages | University | Not in cards | Aug 2025 | ||||
| ITBench-AA Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON. | Artificial Analysis + IBM Research | Terminal tasks | Expert | Not in cards | May 2026 | ||||
| IVISTA Voice Interaction, revised 2023 protocol In-car assistants carry out spoken commands from different seats and answer follow-up requests. Testers check completed actions, including window adjustments and navigation commands. | China Automotive Engineering Research Institute | General assistants | Everyday | Not in cards | Mar 2025 | ||||
| IWSLT 2017 TED Translation Models translate TED talk transcripts between languages. The campaign includes bilingual translation and multilingual translation with unseen language pairs. | IWSLT | Translation | University | Not in cards | |||||
| JailTrickBench Models respond to jailbreak attacks under varied attack and defense settings. A GPT-4 evaluator judges attack success across harmful-request datasets. | HKUST (Guangzhou), Guangzhou HKUST Fok Ying Tung Research Institute and HKUST | Jailbreaks and prompt injection | Expert | Not in cards | Jun 2024 | ||||
| Jam-ALT Models transcribe song lyrics, including punctuation and line breaks. The scorer reports word error rate and separate measures for formatting. | AudioShake | Speech recognition and translation | Everyday | Not in cards | Nov 2023 | ||||
| JAMA Clinical Challenge (JesseLiu) Models choose among four answers to clinical vignettes from JAMA. The OpenMed port combines 1,521 cases and reserves about 5% for evaluation. | JesseLiu | Medicine | Expert | Not in cards | Nov 2024 | ||||
| JamendoLyrics MultiLang Models align written lyrics with singing across 80 songs in English, French, German, and Spanish. The researchers compare predicted and annotated word onsets. | Spotify Research | Speech recognition and translation | Everyday | Not in cards | Jun 2023 | ||||
| JamendoLyrics original Replaced in Jun 2023 | Daniel Stoller and colleagues | Not in cards | Feb 2019 | ||||||
| Jamp Models infer temporal relationships between Japanese sentences. Separate test splits vary the templates, time formats, and time spans. | Tomoki Sugimoto and colleagues | Reasoning in many languages | School | Not in cards | Jun 2023 | ||||
| JaNLI Models judge entailment between Japanese sentences designed to expose shortcuts in language inference. | Hitomi Yanaka and Koji Mineshima | Reasoning in many languages | School | Not in cards | Nov 2021 | ||||
| Japanese LLM Leaderboard (lm-evaluation-harness) Models complete eight Japanese tasks covering reading, reasoning, math, and summarization. The harness groups the individual results. | EleutherAI harness contributors | Reading in many languages | School | Not in cards | Nov 2024 | ||||
| Japanese Tort-case Dataset Models predict whether a Japanese court found a tort from the facts and the parties' arguments. They also label which arguments the court accepted. | Hiroaki Yamada and colleagues | Law | Expert | Not in cards | May 2024 | ||||
| Japanese-English Bilingual Corpus of Wikipedia’s Kyoto Articles Models translate between Japanese and English using parallel Wikipedia passages about Kyoto. | NICT | Translation | School | Not in cards | Oct 2010 | ||||
| JAQKET Models answer Japanese quiz questions. | Tohoku University | Knowledge in many languages | School | Not in cards | |||||
| JASMINE Models replace medical jargon in Japanese sentences with everyday language. The reference sentences come from patients' blogs. | Ehime University and Osaka University | Medicine | University | Not in cards | Mar 2024 | ||||
| Jawaher Models translate Arabic proverbs into English and explain their meanings in English and Arabic. Human raters score translations and explanations on five-point scales. | Mohamed bin Zayed University of Artificial Intelligence and University of British Columbia | Translation | Everyday | Not in cards | Feb 2025 | ||||
| JBLiMP Models compare 331 pairs of Japanese sentences that differ in grammatical acceptability. | Oseki Lab | Reasoning in many languages | Everyday | Not in cards | Jun 2022 | ||||
| JEEM Vision-language models caption images and answer visual questions in four Arabic dialects. The evaluation compares outputs with reference annotations. | Toloka and MBZUAI | Knowledge in many languages | Everyday | Not in cards | Apr 2025 | ||||
| JEMHopQA Models answer Japanese questions that require combining facts across multiple documents and provide the supporting reasoning. | Ai Ishii and colleagues | Reading in many languages | School | Not in cards | Jul 2023 | ||||
| Jester gesture recognition Models classify hand gestures in short videos. The task distinguishes movements with similar appearance but different directions or timing. | TwentyBN | Video understanding | Everyday | Not in cards | Oct 2019 | ||||
| JFinQA Models calculate financial ratios, check consistency, and reason about changes over time using Japanese securities reports. | ajtgjmdjp | Finance | University | Not in cards | Feb 2026 | ||||
| JGLUE Models classify Japanese sentences, compare sentence meanings, and answer reading comprehension and commonsense questions. The authors publish baseline scores for each task. | Yahoo Japan Corporation and Waseda University | Reading in many languages | School | Not in cards | Mar 2022 | ||||
| JGPQA Models answer graduate-level science questions translated into Japanese from GPQA. | LLM-jp | Knowledge in many languages | Expert | Not in cards | Sep 2025 | ||||
| JHumanEval Models implement Python functions from Japanese prompts. Programmers reviewed machine translations of the HumanEval specifications. | Kuramitsu Lab | Function-level problems | School | Not in cards | Sep 2023 | ||||
| Jiminy Cricket Agents play 25 text adventures. Annotations on game states track moral behavior alongside game rewards. | Dan Hendrycks and collaborators | Deception and alignment | Everyday | Not in cards | Oct 2021 | ||||
| JMultiWOZ-TC Models use Japanese dialogue history and past tool results to predict which tools to call next. | LLM-jp | Function calling | Everyday | Not in cards | Dec 2025 | ||||
| JobBERT Job Title Normalization Models match English vacancy titles from Malaysia’s public job board to standardized occupation labels. The authors report reciprocal rank and recall at several cutoffs. | TechWolf and Ghent University | Other professions | University | Not in cards | Sep 2021 | ||||
| Joi-Bench-Task Voice agents use tools while a simulated caller changes a request or interrupts a reply. The evaluator tracks completion, responsiveness, and omitted information. | Tencent | Speech recognition and translation | Everyday | Not in cards | Nov 2026 | ||||
| JSeM Models decide whether a Japanese hypothesis follows from one or more premises. Native speakers provide the reference judgments. | Daisuke Bekki and collaborators | Reasoning in many languages | School | Not in cards | Sep 2019 | ||||
| JSPLIB Agents schedule operations on specified machines. Each job must follow its operation order, and each machine can process only one operation at a time. Shorter schedules score better. | JSPLIB contributors | Tasks from real jobs | University | Not in cards | Apr 2014 | ||||
| K-MHaS Models assign hate speech labels to Korean news comments. A comment can receive more than one label. | Jean Lee and colleagues | Bias, fairness and politics | Everyday | Not in cards | Oct 2022 | ||||
| Kaeru Flip (paprika3d) Agents play an 86-note rhythm game from browser pixels. They must finish the song with at most one miss and leave autoplay off. | jessicayjung | Video games | Everyday | Not in cards | Nov 2026 | ||||
| Kaggle Standardized Agent Exams Agents answer 16 questions on multistep reasoning and handling manipulative prompts. Kaggle publishes an overall score. | Kaggle | Deception and alignment | School | Not in cards | Mar 2026 | ||||
| KairosQA Models answer questions about facts that changed over time, with each item tied to a time period. | Kyutai | Language reasoning | University | Not in cards | Feb 2026 | ||||
| Kazakh Speech Corpus 2 Models transcribe Kazakh broadcasts, parliamentary speeches and podcasts alongside read speech. Some speakers switch between Kazakh and Russian. | ISSAI, Nazarbayev University | Speech recognition and translation | Everyday | Not in cards | Aug 2022 | ||||
| Kazakh Speech Corpus v1 Replaced in Aug 2022 | Not in cards | Sep 2020 | |||||||
| KernelBench-Verified Language models generate CUDA kernels that must match PyTorch outputs and improve runtime. The verifier checks correctness with robust performance baselines. | Meta FAIR and Superintelligence Labs | GPU kernels | Research frontier | Not in cards | Jan 2026 | ||||
| KernelBenchFiltered Agents optimize GPU kernels. METR takes the geometric mean of the speedups across tasks after checking correctness. | METR | GPU kernels | Expert | Not in cards | Feb 2025 | ||||
| KeSpeech Models transcribe Mandarin speech from speakers with regional accents. Qwen compares transcription error against other speech recognizers. | KeSpeech team | Speech recognition and translation | Everyday | Not in cards | |||||
| Keystone-Eval Agents configure Dockerfiles and test entrypoints for arbitrary code repositories, then run the repositories' tests. | Imbue | Fixing real repositories | Expert | Not in cards | Apr 2026 | ||||
| KgCLUE Models answer Chinese questions using a knowledge graph. | CLUE benchmark team | Questions that need search | School | Not in cards | |||||
| Khmer Handwritten Dataset (4.2k) Models transcribe line images from a handwriting collection that includes Khmer and English text. | Royal University of Phnom Penh | Documents, charts and text in images | Everyday | Not in cards | Oct 2024 | ||||
| Kiji PII Detection Models locate and label personal information in multilingual text. Dataiku publishes a test split, and independent evaluators compare predicted spans with its annotations. | Dataiku | Following precise instructions | Everyday | Not in cards | Feb 2026 | ||||
| Kili LLM Reasoning Benchmark Models answer language, mathematics, and reasoning questions at five proficiency levels. | Kili Technology | Language reasoning | University | Not in cards | |||||
| KiloGram Models choose which of ten tangram pictures matches a description. The descriptions can name the whole shape or explain what its pieces represent. | Cornell University, New York University, Allen Institute for AI and Princeton University | Spatial reasoning and counting | Everyday | Not in cards | Nov 2022 | ||||
| KilometerVision Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures. | Google DeepMind (Perception Test team) | Spatial reasoning and counting | Everyday | Not in cards | Jun 2026 | ||||
| Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks. | Moonshot AI | Fixing real repositories | Expert | Not in cards | |||||
| KITTI depth prediction Models estimate scene depth from road images. The evaluation compares model predictions with depth measurements collected from a moving vehicle. | Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago | Spatial reasoning and counting | School | Not in cards | |||||
| Knowledge Cutoff Benchmark Models answer questions about dated events. A judge checks direct answers against known outcomes. A separate multiple-choice test uses an answer key. | Apoorv Saxena | Short factual questions | Everyday | Not in cards | Jul 2026 | ||||
| KnowledgeGraphEval 2026 Models label nested entities in Arabic text from ten domains. A separate test asks models to classify relations between marked pairs of entities. | KnowledgeGraphEval organizers | Reading in many languages | University | Not in cards | |||||
| KoBEST Models do five Korean tasks designed by professional linguists: BoolQ, COPA, WiC, HellaSwag and SentiNeg. All data is human-annotated. | SK Telecom and collaborators (Kim, Jang et al.) | Reasoning in many languages | Everyday | Not in cards | Apr 2022 | ||||
| Kompetencer Models classify skill and knowledge phrases from Danish job ads using a European occupational taxonomy. The authors report weighted macro F1. | IT University of Copenhagen and LMU Munich | Other professions | University | Not in cards | Apr 2022 | ||||
| KoMT-Bench Models follow multi-turn instructions in Korean on a translated and culturally adapted version of MT-Bench, reviewed by expert linguists. | LG AI Research | Chat and instructions in many languages | Everyday | Not in cards | Aug 2024 | ||||
| Kontext Bench Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency. | Black Forest Labs | Video generation | Everyday | Not in cards | Jun 2025 | ||||
| Kor-Phishing Models classify web requests and links as malicious or benign. Korea's AI Safety Institute reports accuracy separately for each group of 100 items. | Korea AI Safety Institute | Secure code and review | University | Not in cards | Dec 2025 | ||||
| KORA Apps Simulated children use consumer AI apps end to end, with each app system prompt, filters and safeguards active. Scores cover online, physical and sexual, psycho-social and learning risks. | KORA (non-profit) | Wellbeing and sycophancy | Everyday | Not in cards | Jul 2026 | ||||
| Korean Canonical Legal Benchmark Models answer Korean legal questions with supporting precedents. The benchmark includes multiple-choice questions and essays with individual grading criteria. | Hongseok Oh and colleagues | Law | University | Not in cards | Oct 2025 | ||||
| Korean National Licensing Examination for Korean Medicine Doctors 2022 Models answer 340 questions from the Korean medicine licensing exam. Researchers remove the images, transcribe tables, and compare prompt variants against the answer key. | Gachon University and Pusan National University | Medicine | Expert | Not in cards | Mar 2023 | ||||
| KPI-EDGAR Models identify key performance indicators and related values in financial reports. The original annotations also link entities within the text. | Tobias Deußer and colleagues | Finance | University | Not in cards | Oct 2022 | ||||
| KSAA 2026 Automatic Diacritization of Speech Dictation Models restore Arabic vowel marks from a recording and its transcript. Organizers rank systems by word error rate with case endings and unmarked characters included. | King Salman Global Academy for Arabic Language and collaborators | Speech recognition and translation | School | Not in cards | Dec 2025 | ||||
| KuralHub Models classify emotions in speech from 29 languages. | Luxshan Thavarasa and colleagues | Audio understanding and voice | Everyday | Not in cards | Feb 2025 | ||||
| KurdiSent Models predict the sentiment expressed in Kurdish text. | Soran Badawi and colleagues | Reading in many languages | Everyday | Not in cards | |||||
| Kyoto University Commonsense Inference Dataset Models select a plausible continuation for a Japanese passage by inferring the relationship between events. | Kyoto University | Reasoning in many languages | Everyday | Not in cards | Nov 2020 | ||||
| Kyoto University Wikipedia Annotated Corpus Models analyze Japanese Wikipedia text. The harness tests reading prediction, names, syntactic dependencies, predicate arguments, and coreference. | Kyoto University | Reading in many languages | University | Not in cards | Oct 2022 | ||||
| LAB: Contracts 500 in-house contracting tasks (drafting, redlines, issues lists, escalation memos) across 10 contract categories. All-pass expert-rubric grading. | Harvey | Law | Expert | Not in cards | Jun 2026 | ||||
| LAB: Diligence Agent researches a synthetic M&A virtual data room (thousands of documents, up to ~80M tokens) and drafts a diligence memo scored against planted-issue rubric criteria. | Harvey | Law | Expert | Not in cards | Jul 2026 | ||||
| LAB: Law Firm Knowledge Agents search a simulated law firm’s matter files and deliver work that satisfies legal task criteria. | Harvey | Law | Expert | Not in cards | Aug 2026 | ||||
| LAB: US Case Law Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings. | Harvey | Law | Expert | Not in cards | Oct 2026 | ||||
| LAD2000 Models identify unusual events in videos and score when they happen. The test contains 280 videos with unusual events and 280 without them. | Jiangxi University of Finance and Economics | Video understanding | Everyday | Not in cards | May 2021 | ||||
| Lahjoita puhetta ASR test Models transcribe spontaneous Finnish speech on the official ten-hour test. Human transcripts supply the reference words. | Aalto University and University of Helsinki | Speech recognition and translation | Everyday | Not in cards | Mar 2022 | ||||
| Lakh MIDI MASD Style Classification Models identify which of 25 styles apply to a MIDI file. The released mapping labels 17,785 files, and each file can have multiple labels. | Universitat Pompeu Fabra and University of Helsinki | Audio understanding and voice | School | Not in cards | Sep 2017 | ||||
| Lakh MIDI Top-MAGD Genre Classification Models identify which of 13 genres apply to a MIDI file. The released mapping labels 22,535 files, and each file can have multiple labels. | Universitat Pompeu Fabra and University of Helsinki | Audio understanding and voice | School | Not in cards | Sep 2017 | ||||
| LAMBADA OpenAI Multilingual Models predict the last word in short narrative passages. The harness includes English and four machine-translated test sets. | EleutherAI | Reading in many languages | Everyday | Not in cards | |||||
| Lamina Product Bench Agents build four small applications under three instructions: code directly, plan first, or use Lamina skills. A model judge grades each submission. | Lamina | Building whole projects | University | Not in cards | Jul 2026 | ||||
| LaminaBench v1 Agents implement the same product brief with and without Lamina skills. The package README specifies ten briefs across five categories and 20 paired Harbor tasks. | Lamina | Building whole projects | University | Not in cards | Jul 2026 | ||||
| LangChain Benchmarks Agents complete tool use, retrieval and extraction tasks with reproducible LangSmith datasets and task-specific graders. | LangChain | General assistants | Expert | Not in cards | Nov 2023 | ||||
| Language-Table A simulated robot moves colored blocks in response to natural language instructions. The benchmark measures success and success weighted by path length across five task families and 696 task variations. | Robotics at Google | Simulated robots | Everyday | Not in cards | Oct 2022 | ||||
| Large Movie Review Dataset Models classify an IMDb movie review as positive or negative. | Andrew Maas and colleagues | Reading comprehension | Everyday | Not in cards | Jun 2011 | ||||
| Large-Vocabulary Video Instance Segmentation Models detect, segment, and track object instances across video frames. The benchmark includes categories outside the training vocabulary. | LV-VIS team | Video understanding | Everyday | Not in cards | Apr 2023 | ||||
| LARYBench Models answer expert questions in the LARYBench dataset released by Meituan LongCat. | Meituan LongCat Team | Expert-written questions | Expert | Not in cards | Mar 2026 | ||||
| Latin Synonym Selection Benchmark Models choose the closest synonym of a Latin lemma from four options. The released file contains 2,759 questions. | LiLa, Università Cattolica del Sacro Cuore | Reading in many languages | University | Not in cards | Oct 2019 | ||||
| LayMaker MTSamples Simplification Models rewrite medical transcription sentences for patients. Clinicians supplied the reference versions and kept sentences they already considered clear. | Babylon Health | Medicine | University | Not in cards | Dec 2021 | ||||
| LCBench v0.1 Models solve English and Chinese LeetCode contest problems. Tests check the submitted programs. | OpenCompass | Competitive programming | Competition | Not in cards | May 2024 | ||||
| LeanDojo Benchmark Models prove theorems from Lean's mathlib library inside the Lean proof assistant, which checks every proof. The benchmark has 98,734 theorems and proofs with premise annotations. A novel-premises split tests whether a prover can use lemmas it never saw in training. | Caltech, NVIDIA and collaborators (LeanDojo) | Formal proofs | University | Not in cards | Jun 2023 | ||||
| LedgerBench-100 Agents perform 100 corporate finance tasks through 66 tools in a shared enterprise system. SQL checks grade the changes they make. | Blobfish AI | Finance | Expert | Not in cards | Aug 2026 | ||||
| Leeds Sports Pose Extended Models locate body joints in photographs of athletes. Meta compares the fraction of predicted joints within a distance threshold. | Sam Johnson and Mark Everingham | Spatial reasoning and counting | School | Not in cards | Jun 2011 | ||||
| Legal Information Retrieval Benchmark Search APIs retrieve the official document behind legal research requests. Exact URL matching scores whether the official page appears first or in the top ten. | Firecrawl | Law | Expert | Not in cards | Nov 2026 | ||||
| Legal Opinion Sentiment Classification Models classify the sentiment expressed in passages from legal opinions. Expert annotations provide the reference labels. | Gathika Ratnayaka and colleagues | Law | University | Not in cards | Oct 2020 | ||||
| Legal World sample Agents act as a legal assistant in an anonymized law practice. Four tasks grade the drafts, mail, calendar changes, and files they leave behind. | doe | Law | Expert | Not in cards | Nov 2026 | ||||
| LegalSupport Models read a legal claim and two court case summaries. They choose the case that most directly supports the claim. | Neel Guha and Stanford CRFM | Law | University | Not in cards | |||||
| LEVIR-CC Models compare aligned aerial images from Texas and describe what changed. Five annotators wrote reference captions for each of the 1,929 test pairs. | Chenyang Liu, Rui Zhao, Hao Chen, Zhengxia Zou, and Zhenwei Shi | Detection and captioning | Everyday | Not in cards | Apr 2022 | ||||
| LEVIR-CDC Models describe changes between aerial images and mark changed buildings. The authors paired LEVIR-CC captions with binary building masks. | Jingye Shi and colleagues | Detection and captioning | Everyday | Not in cards | Oct 2024 | ||||
| LexGLUE Models solve a suite of English legal text tasks drawn from court cases, EU law, contracts and terms of service. Most tasks are classification, and one asks for the holding of a US case. Each task reports micro and macro F1. | University of Copenhagen and collaborators | Law | Expert | Not in cards | Oct 2021 | ||||
| LEXTREME Models analyze legal text across languages. Tasks include predicting document labels, identifying legal arguments, and labeling words within documents. | Joel Niklaus and colleagues | Law | University | Not in cards | Jan 2023 | ||||
| LGEB Models generate summaries and questions from source passages. | CLUE benchmark team | Reading comprehension | School | Not in cards | |||||
| LightGenBench Models predict which parts of a 3D surface emit light and what color they emit. The authors compare generated emission with reference assets. | Dongchen Yang and colleagues | Image generation and editing | University | Not in cards | Oct 2026 | ||||
| Lights Out 3×3 Models toggle lights on a three-by-three grid until every light is off. The dataset reserves 30 boards for testing and rewards efficient solutions. | candukuri | Puzzles and chess | School | Not in cards | Sep 2025 | ||||
| LIMIT Models retrieve specific pairs of documents from a shared collection using embeddings. | Orion Weller and colleagues | Questions that need search | School | Not in cards | Aug 2025 | ||||
| LIMMITS 2025 Speech models synthesize text in four languages. Listeners rate naturalness and voice similarity. Separate tracks also measure streaming latency, codec bitrate, or both. | Indian Institute of Science and LIMMITS organizers | Audio understanding and voice | Everyday | Not in cards | Aug 2024 | ||||
| LIMMITS 2024 Replaced in Aug 2024 | Not in cards | Sep 2023 | |||||||
| Linear Algebra Done Right (Axler Exercises) Models solve 186 linear algebra tasks on 25 topics drawn from chapters one through nine of Axler’s textbook. Exact rational arithmetic checks their answers. | nair-akash | Word problems | University | Not in cards | Aug 2026 | ||||
| LineEX Models extract labels and numerical series from scientific line charts. The hub port uses a 20,000-image test split and grades the structured output. | IIT Gandhinagar | Documents, charts and text in images | University | Not in cards | Jul 2022 | ||||
| LitBench Judges predict human preferences across 2,480 held-out story pairs. The hub port instead samples the training corpus and adds self-critique. | SAA Lab | Taste and style | University | Not in cards | Jul 2025 | ||||
| LitSearch Models retrieve relevant research papers in response to questions about the scientific literature. | Anirudh Ajith and colleagues | Questions that need search | Expert | Not in cards | Jul 2024 | ||||
| LiveMACE Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns. | Fudan University | Finance | Expert | Not in cards | Oct 2026 | ||||
| LiveMath Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures. | Google DeepMind | Competition problems with answers | Competition | Not in cards | Sep 2025 | ||||
| LlamaTouch Agents carry out requests in Android apps. The evaluator checks whether each run visited the required app or system states. | Beijing University of Posts and Telecommunications and Tsinghua University | Phones | Everyday | Not in cards | Apr 2024 | ||||
| LLaVA-Bench (In-the-Wild) Multimodal assistants answer open questions about a small set of diverse real images. A GPT-4 judge scores each answer relative to a reference answer. | UW-Madison and Microsoft Research (LLaVA team) | General visual questions | Everyday | Not in cards | Apr 2023 | ||||
| LLM Economist Models set tax rates or choose working hours in a simulated economy. Researchers compare the social welfare achieved under different tax policies. | Princeton University and Salesforce Research | Running a business | University | Not in cards | Jul 2025 | ||||
| LLM Training Puzzles Models solve eight coding puzzles on distributed neural network training, including optimizer state and pipeline parallelism. The hub evaluates each solution in a sandbox. | Sasha Rush | Machine learning engineering | Expert | Not in cards | Jun 2023 | ||||
| llm-jp-eval Models answer Japanese language understanding questions, translate passages, and follow instructions. The harness reports scores for individual datasets and groups of tasks. | LLM-jp | Reasoning in many languages | School | Not in cards | Oct 2023 | ||||
| LLM-NERRE materials extraction Models read materials papers and link dopants to host materials, guest species to metal-organic frameworks, and crystal structures to chemical formulas. | Lawrence Berkeley National Laboratory and University of California, Berkeley | Lab and data work | University | Not in cards | Dec 2022 | ||||
| LLMEval-Med Models answer clinical questions drawn from patient records and expert scenarios. A model grades answers against physician-developed checklists. | Ming Zhang and colleagues | Medicine | Expert | Not in cards | Jun 2025 | ||||
| LM-SynEval Models compare English sentences that differ in grammaticality. The test measures preferences for correct agreement and other syntactic dependencies. | Rebecca Marvin and colleagues | Language reasoning | Everyday | Not in cards | Oct 2018 | ||||
| LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. | Google DeepMind | Embodied reasoning | University | Not in cards | Dec 2024 | ||||
| LMSYS Chatbot Arena Human Preference Predictions Models predict which answer a user preferred in an anonymous pair. Kaggle scores the submitted probabilities by log loss. | LMSYS and Kaggle | Overall chat quality | University | Not in cards | May 2024 | ||||
| LoCoMo Models answer questions, summarize events and continue dialogue over very long multi-session chats of about 300 turns and 9K tokens, up to 35 sessions. Answers are scored against references. | Snap Research, UNC Chapel Hill and USC (Maharana et al.) | Finding facts in long inputs | Everyday | Not in cards | Feb 2024 | ||||
| LogiQA Models answer reading comprehension questions that test logical reasoning, taken from Chinese civil service exams and translated to English. | Liu et al. (LogiQA authors) | Language reasoning | University | Not in cards | Jul 2020 | ||||
| Long Context Navigator Agents answer questions that link facts across a synthetic document of about 32,000 tokens. They must read, compress, or discard information within a 600-token working context. | manasganti | Reasoning over long documents | University | Not in cards | Oct 2026 | ||||
| Long Range Arena Transformers solve six long-sequence tasks covering list operations, text, retrieval, image, path and path crossing. | Google Research and collaborators | Finding facts in long inputs | Expert | Not in cards | Jan 2021 | ||||
| Long-document QA (Kensho) long-document-qa Models answer questions by reasoning over multi-hundred-page financial filings and other long documents. | Kensho Technologies, S&P Global | Finance | Expert | Not in cards | Oct 2024 | ||||
| Finance Fundamentals finance-fundamentals Replaced in Oct 2024 | University | Not in cards | Apr 2024 | ||||||
| Long-term Video Object Segmentation v2 Models track object masks through long videos. The evaluation tests whether they can recover an object's identity after an extended absence. | Lingyi Hong and colleagues | Video understanding | Everyday | Not in cards | Apr 2024 | ||||
| LongBench Chinese Models answer questions and complete other long-context tasks in the original Chinese headline table. The score is its published average on a percentage scale. | THUDM (Tsinghua University) | Reasoning over long documents | University | Not in cards | Aug 2023 | ||||
| LongLeader Models answer a unified suite of long-context tasks across multiple context lengths and abilities. Accuracy tables compare language models and context-extension methods. | Amazon Science | Long context | University | Not in cards | Jan 2025 | ||||
| LongPuzzleBench GUI agents solve 114 visual puzzle levels across six games through native interface actions. | LongPuzzleBench authors | Puzzles and chess | Everyday | Not in cards | Sep 2026 | ||||
| LoTTE Models retrieve passages that answer questions about less common topics. | Keshav Santhanam and colleagues | Questions that need search | School | Not in cards | Dec 2021 | ||||
| LoveDA Models label land cover in aerial images of urban and rural areas. The benchmark supports comparisons across these two settings. | Junjue Wang and colleagues | Detection and captioning | University | Not in cards | Oct 2021 | ||||
| LP-TTS-Bench Models synthesize speech from extended recordings of a speaker. The benchmark measures how well they reproduce timing, emphasis, and intonation on 2,728 held-out utterances. | The Chinese University of Hong Kong, Shenzhen and Amphion Technology | Speech recognition and translation | Everyday | Not in cards | Oct 2026 | ||||
| LVIS Models locate objects and outline each instance across a large vocabulary of categories. Meta reports box and mask average precision. | Agrim Gupta and colleagues | Detection and captioning | Everyday | Not in cards | Aug 2019 | ||||
| LVSum Models summarize long videos with timestamped segments, relevance scores and descriptions using video frames, transcripts or both. | Apple | Long videos | Expert | Not in cards | Apr 2026 | ||||
| M3-Bench Multimodal agents answer questions about long videos while using long-term memory and multimodal retrieval. | ByteDance Seed | Long videos | Expert | Not in cards | Aug 2025 | ||||
| M³AV Speech Recognition Models transcribe academic lectures. The authors measure word errors across the transcript and separately for rare words. Slide text can provide context. | Shanghai Jiao Tong University and collaborators | Speech recognition and translation | University | Not in cards | Mar 2024 | ||||
| M4Singer Models transcribe lyrics from singing voices. Qwen compares recognition systems on the M4Singer recordings. | M4Singer team | Speech recognition and translation | Everyday | Not in cards | |||||
| MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. The score is normalized game reward. Annotations also score power-seeking, disutility and ethical violations. | UC Berkeley and Center for AI Safety | Deception and alignment | Everyday | Not in cards | Apr 2023 | ||||
| MacroLens Models use company and economic data to make financial estimates under different scenarios. The authors compare forecasting models with language models and simple baselines. | Patara Trirat and colleagues | Finance | University | Not in cards | Jun 2026 | ||||
| MAD-Bench Multimodal models answer image questions containing deceptive prompt information about objects, counts and spatial relationships. | Apple | General visual questions | University | Not in cards | Feb 2024 | ||||
| MAD-Eval Models write descriptions for scenes from ten movies. One set of references names characters. Another replaces their names with the word “SOMEONE.” | University of Oxford, École des Ponts and Shanghai Jiao Tong University | Video captioning | Everyday | Not in cards | Mar 2023 | ||||
| MADASR 2.0 Models transcribe read speech and spontaneous conversations across eight Indian languages. The organizers rank each track by character error and also score language and dialect identification. | Indian Institute of Science and MADASR organizers | Speech recognition and translation | Everyday | Not in cards | Apr 2025 | ||||
| MADASR 2023 1.0 Replaced in Apr 2025 | Not in cards | May 2023 | |||||||
| MADE 1.0 Models identify drugs, indications, and adverse events in cancer care notes. They link each drug to its attributes and effects. The challenge scores extraction, linking, and the combined task. | University of Massachusetts | Medicine | University | Not in cards | Nov 2017 | ||||
| MAESTRO Piano Transcription Models transcribe competition piano recordings into notes. The evaluator compares predictions with MIDI events captured by the instrument. Each composition appears in only one split. | Google Magenta and the International Piano-e-Competition | Audio understanding and voice | Expert | Not in cards | Oct 2018 | ||||
| MAFAND-MT Models translate news text in a benchmark covering 21 African languages. | David Ifeoluwa Adelani and colleagues | Translation | School | Not in cards | Jul 2022 | ||||
| Magentic Marketplace Agents search for sellers, exchange proposals and make purchases. The simulation measures how much value customers receive after paying for their purchases. | Microsoft Research | Running a business | Everyday | Not in cards | Oct 2025 | ||||
| MagnaTagATune Models predict descriptive tags for music clips. Qwen compares their predictions with human annotations using micro F1. | Edith Law and colleagues | Audio understanding and voice | Everyday | Not in cards | |||||
| Make Data Count: Finding Data References Models find dataset identifiers in scientific papers and classify each reference as data the authors created or reused. Kaggle scores the labeled references with F1. | Make Data Count and DataCite | Reading comprehension | University | Not in cards | Jun 2025 | ||||
| Malaysian Sign Language Medical Dataset Models identify Malaysian Sign Language terms used in health care. The training and test lists share all seven signer names. | Swinburne University of Technology Sarawak and Sarawak Society for the Deaf | Video understanding | Everyday | Not in cards | Jul 2023 | ||||
| Málrómur Models transcribe Icelandic sentences that volunteers recorded on smartphones. Human listeners checked the reference text against the recordings. | Reykjavík University and the Icelandic Centre for Language Technology | Speech recognition and translation | Everyday | Not in cards | Aug 2014 | ||||
| Maltese News Categories Models assign one or more topic labels to Maltese news articles. | Amit Kumar Chaudhary and colleagues | Reading in many languages | School | Not in cards | May 2024 | ||||
| MammalNet Models identify mammals and their behavior in videos. The benchmark covers 173 mammal categories and 12 behaviors, including nursing and hunting. | Jun Chen and colleagues | Video understanding | School | Not in cards | Mar 2023 | ||||
| MAMUBench Agents represent different users who share a limited resource. The benchmark compares their combined outcome with one agent coordinating for everyone. | Anthropic Fellows Program and collaborators | Multi-agent games | Everyday | Not in cards | Oct 2026 | ||||
| ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. | Chongkai Gao and colleagues | Simulated robots | Everyday | Not in cards | Nov 2026 | ||||
| ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. | X Square Robot | Real robots | Everyday | Not in cards | Mar 2026 | ||||
| ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. | UC San Diego (Hao Su Lab) + Hillbot | Simulated robots | Everyday | Not in cards | Oct 2024 | ||||
| Mantic vs Polymarket backtest Point-in-time backtest of Mantic's AI forecasts against prices on 4,139 Polymarket markets at volume snapshots. Scored by Brier score and calibration. | Mantic | Real-world events | Expert | Not in cards | Sep 2026 | ||||
| Manually-Reviewed Agentic Labeled Transcripts Monitors classify agent transcripts for reward hacking and sandbagging. METR compares their predictions with human labels using the area under the receiver operating characteristic curve. | METR | Deception and alignment | Expert | Not in cards | Oct 2025 | ||||
| Many Labs 2 text-based replications Models answer questions as participants in 14 psychology experiments. Researchers compare the effects with human replications and report the share of findings the model reproduces. | Peter S. Park, Philipp Schoenegger, and Chongyang Zhu | Social simulation | Everyday | Not in cards | Feb 2023 | ||||
| MAPA EUR-Lex Models mark sensitive entities in EU legal texts. MAPA annotates general and fine-grained categories. The multilingual conversion covers 21 languages. | Multilingual Anonymisation for Public Administrations consortium | Law | University | Not in cards | |||||
| MapAnything dense multi-view reconstruction Models reconstruct a scene from multiple images. The benchmark compares point geometry, metric scale, depth, and camera poses with reference measurements. | Meta | Spatial reasoning and counting | School | Not in cards | Sep 2025 | ||||
| MapAnything single-view calibration Models infer camera intrinsics from a single image. The benchmark measures angular error between predicted camera rays and reference rays. | Meta | Spatial reasoning and counting | School | Not in cards | Sep 2025 | ||||
| MAPS Piano Transcription Models transcribe piano recordings into notes. A common test uses 60 Disklavier recordings from two microphone setups and checks predicted pitches and timing against MIDI. | Télécom ParisTech | Audio understanding and voice | Expert | Not in cards | |||||
| Markov Blanket Discovery (shinpaku) Models write a function that identifies the variables in a target’s Markov blanket from binary data. The evaluator measures F1 against known variable sets. | shinpaku | Machine learning engineering | Expert | Not in cards | Aug 2026 | ||||
| MasakhaNER 2.0 Models identify named entities in human-annotated text across 20 African languages. | David Ifeoluwa Adelani and colleagues | Reading in many languages | School | Not in cards | Oct 2022 | ||||
| MasakhaNER 1.0 Replaced in Oct 2022 | Not in cards | Mar 2021 | |||||||
| MasakhaNEWS Models predict the topic of news articles written in African languages. | David Ifeoluwa Adelani and colleagues | Reading in many languages | Everyday | Not in cards | Apr 2023 | ||||
| MASSIVE Models identify an utterance’s intent and label its argument slots across 51 languages. The held-out test reports intent accuracy, slot-filling F1, and exact match accuracy for the complete prediction. | Amazon Alexa AI | Reading in many languages | Everyday | Not in cards | Apr 2022 | ||||
| Massive Visualized IR Benchmark Models retrieve screenshots that match a search query. The benchmark compares performance across four retrieval task groups. | Beijing Academy of Artificial Intelligence | Documents, charts and text in images | Everyday | Not in cards | Feb 2025 | ||||
| Massively Multilingual Speech labeled corpus Models transcribe recorded readings across languages. The evaluation compares predictions with reference transcripts using character error rate. | Meta | Speech recognition and translation | School | Not in cards | May 2023 | ||||
| Mastermind Information Gain Models choose a Mastermind move from earlier guesses and feedback. The evaluator measures information gain for individual moves and success over complete games. | peterN908 | Puzzles and chess | School | Not in cards | Sep 2025 | ||||
| MatBERT gold nanoparticle entity recognition Models mark particle names and words describing their shapes in passages from gold nanoparticle papers. The evaluation scores both types of text span. | Lawrence Berkeley National Laboratory and UC Berkeley | Lab and data work | University | Not in cards | Oct 2021 | ||||
| Materials Science Procedural Text Corpus Models identify operations in materials synthesis procedures and link each operation to its materials, instruments, and conditions. Experts annotated 230 procedures. | University of Massachusetts Amherst and Massachusetts Institute of Technology | Lab and data work | University | Not in cards | Nov 2018 | ||||
| Mathematical Problem Repository for AlphaEvolve Agents generate programs that construct mathematical objects or improve known bounds. The repository documents 67 problems and provides verification code for many of them. | Google DeepMind | Research mathematics | Research frontier | Not in cards | Nov 2025 | ||||
| MathOlympiadBench Models produce Lean 4 proofs for 360 human-checked formalizations of olympiad problems. Lean checks each proof. | Goedel-LM | Formal proofs | Competition | Not in cards | Aug 2025 | ||||
| MatSci-NLP Models read text from materials science papers and complete seven tasks, including identifying synthesis actions and linking measurements to experiments. The authors used 99% of examples for testing. | Université de Montréal, Mila, and Intel Labs | Lab and data work | University | Not in cards | May 2023 | ||||
| MAUD Models classify provisions in merger agreements into expert-defined legal categories. The benchmark compares predictions with lawyer annotations and reports precision-recall area and micro and macro F1. | ETH Zurich, Harvard University, The Atticus Project and others | Law | Expert | Not in cards | Jan 2023 | ||||
| MAWARITH v2026 Models identify eligible heirs and calculate each share of an estate from an Arabic case. The MIR-E score gives credit for intermediate decisions. | Hamad Bin Khalifa University and Nazarbayev University | Law | University | Not in cards | Dec 2025 | ||||
| QIAS 2025 Islamic Inheritance Reasoning Replaced in Dec 2025 | QIAS organizers | Not in cards | May 2025 | ||||||
| MAWPS Models solve one-unknown arithmetic and algebra word problems drawn from several earlier collections. Each problem has an equation and a numeric answer, and the final answer is scored by exact match. | University of Washington and collaborators (Koncel-Kedziorski, Roy, Amini, Kushman, Hajishirzi) | Word problems | School | Not in cards | Jun 2016 | ||||
| Maydan HF RL Sampler Agents reconcile synthetic Saudi social insurance contributions across ten payroll tasks. SQLite checks the final records and audit log. | Maydan Labs | Documents and spreadsheets | Expert | Not in cards | Oct 2026 | ||||
| MBench Models generate human motion from 450 prompts. The evaluation measures motion quality and uses a vision-language model to judge prompt matching and generalization. | ViMoGen team at Nanyang Technological University, SenseTime Research and collaborating institutions | Other | Everyday | Not in cards | Oct 2025 | ||||
| MBTI Post Classification (Kunumi) Models predict an author’s four-letter Myers-Briggs type from their posts. The port holds out one quarter of the rows and gives partial credit for matching letters. | Kunumi | Reading comprehension | Everyday | Not in cards | Sep 2025 | ||||
| MC-TACO Models judge plausible answers to questions about when events happen, how often they recur and how long they last. | Ben Zhou and colleagues | Commonsense | Everyday | Not in cards | Sep 2019 | ||||
| MCP-SIM Agents turn physics problems into FEniCS simulations. The tasks include heat flow, elastic deformation, and fracture. The authors count solved tasks. | Donggeun Park, Hyunbin Moon, and Seunghwa Ryu | Scientific code | University | Not in cards | Jun 2025 | ||||
| MdCoEditBench Agents edit selected Markdown in the WorkBuddy editor alongside a user. Tasks include restoring a version, resolving an edit conflict, and continuing existing text. | Tencent | Documents and spreadsheets | University | Not in cards | Nov 2026 | ||||
| MeasEval Models extract quantities and units from scientific paragraphs and link them to measured properties, entities, and qualifiers. The competition averages scores across nine components. | Elsevier Labs and the University of Amsterdam | Lab and data work | University | Not in cards | Jul 2020 | ||||
| MECCANO Action Recognition Models identify an assembly action from a first-person video clip of someone building a toy motorcycle. The answer key has 61 action classes. | University of Catania and collaborators | Video understanding | Everyday | Not in cards | Oct 2020 | ||||
| Med-EASi Models rewrite English medical passages in plain language. The references mark replacements, explanations, insertions, and deletions. | Stanford University, University of Zurich, and Cornell University | Medicine | University | Not in cards | Feb 2023 | ||||
| MedAlign Models follow clinician-written instructions using a patient’s electronic medical record. Clinicians evaluate the responses. | Scott L. Fleming and colleagues | Medicine | Expert | Not in cards | Aug 2023 | ||||
| MedBench (Russian closed-test platform) Closed-test platform Models answer Russian medical questions and classify relationships between clinical statements. The platform also scores electrocardiogram labels on a separate task. | MedBench team | Medicine | University | Not in cards | Nov 2022 | ||||
| RuMedBench Original Replaced in Nov 2022 | Sber Artificial Intelligence Laboratory | Expert | Not in cards | Jan 2022 | |||||
| MedDialog Models generate a clinician’s response from a patient description. The harness also evaluates summaries of medical questions using reference text. | Guangtao Zeng and colleagues | Medicine | Expert | Not in cards | Nov 2020 | ||||
| MEDDOPROF Models find occupations and employment status in Spanish clinical notes. They identify whose occupation each mention describes and assign an occupational or clinical code. | Barcelona Supercomputing Center | Medicine | University | Not in cards | Dec 2020 | ||||
| MedGemma lab report conversion Models convert lab reports to structured JSON. Google reports macro and micro F1 on two collections of clinical PDFs and a collection of synthetic reports. | Documents, charts and text in images | School | Not in cards | Jan 2026 | |||||
| MedGenVidQA 2026 Visual Answer Localization Models mark the start and end of the video segment that answers a medical question. The organizers created 80 test examples from professional medical videos. | U.S. National Library of Medicine | Medicine | University | Not in cards | Jan 2026 | ||||
| Media Bias Identification Benchmark Models classify text in eight tasks, including linguistic bias, political bias, and hate speech. The benchmark reports F1 across pooled examples and separately by source dataset. | Media Bias Group | Bias, fairness and politics | Everyday | Not in cards | Feb 2023 | ||||
| Media Frames Corpus Models identify the main frame of a US news article, such as economic consequences or public opinion. Researchers compare predictions with human labels. | Dallas Card, Amber Boydstun, Justin Gross, Philip Resnik, and Noah Smith | Reading comprehension | School | Not in cards | Jul 2015 | ||||
| Medical Paper-Blog Dataset Models rewrite a sentence from a medical paper for a general reader. The reference sentence comes from a medical blog. | International Institute of Information Technology Hyderabad | Medicine | University | Not in cards | May 2020 | ||||
| Medical Prescription Selection (lm-evaluation-harness) Models read a diagnosis or complaint and choose the recorded prescription. The two task variants use different distractors. | EleutherAI lm-evaluation-harness contributors | Medicine | Expert | Not in cards | Mar 2025 | ||||
| MedICaT Models locate panels within a biomedical figure and match each panel to the relevant words in its caption. The original test scores word overlap. | Allen Institute for AI and collaborators | Documents, charts and text in images | University | Not in cards | Oct 2020 | ||||
| MedicationQA Models answer consumer questions about medications. The original corpus pairs questions with expert answers, and the hub evaluation grades answer accuracy and completeness. | US National Library of Medicine | Medicine | University | Not in cards | Mar 2019 | ||||
| MEDIQA 2019 Question Answering Models answer consumer health questions or assess candidate answers. The shared task evaluates the relevance and quality of answers to medical questions. | Asma Ben Abacha and colleagues | Medicine | Expert | Not in cards | Aug 2019 | ||||
| MEDIQA-SYNUR Models extract structured observations from synthetic nursing dictations and map them to a clinical vocabulary. The hub uses the development split by default and supports the test split. | MEDIQA organizers | Medicine | Expert | Not in cards | Dec 2025 | ||||
| MedMentions Models identify biomedical concepts in PubMed abstracts and link text spans to identifiers in the Unified Medical Language System. | Chan Zuckerberg Initiative | Genomics and life sciences | University | Not in cards | May 2018 | ||||
| MedPhysBench Knowledge Questions Models answer board-style medical physics questions covering radiation oncology, dosimetry, imaging, and radiation protection. Version 0.2.0 contains 412 questions, including 115 test items. | omar | Medicine | Expert | Not in cards | Jan 2026 | ||||
| MedProcNER Models identify procedures in Spanish clinical cases, link them to SNOMED CT codes, and list the codes for each case. | Barcelona Supercomputing Center | Medicine | University | Not in cards | Jan 2023 | ||||
| MedPT Models classify Brazilian Portuguese medical questions by specialty. The original evaluation tests 20 classes, while the hub creates a split after deduplication and label selection. | Fernanda Bufon Farber and colleagues | Medicine | University | Not in cards | Nov 2025 | ||||
| MedQuAD Systems answer medical questions using a collection of 47,457 question-answer pairs from NIH websites. The OpenMed port uses a model judge on a subset. | Asma Ben Abacha and Dina Demner-Fushman | Medicine | University | Not in cards | Jan 2019 | ||||
| MedReadMe Models rate how easy a medical sentence is to understand. Researchers compare the ratings with judgments from readers. | Georgia Institute of Technology | Medicine | Everyday | Not in cards | May 2024 | ||||
| MedSecId Models identify section boundaries and assign a section type in clinical notes. The authors annotated 2,002 notes and reserved 10% for testing. | University of Illinois at Chicago | Medicine | University | Not in cards | Oct 2022 | ||||
| MedText Models suggest diagnoses or treatments for textbook-style patient presentations. The harness compares their text with the recorded completion. | BI55 (MedText dataset) | Medicine | Expert | Not in cards | Jul 2023 | ||||
| MedVidQA 2023 Medical Instructional Question Generation Models write a question that a supplied medical video segment answers. They can use both the images and the subtitles. | U.S. National Library of Medicine and National Institute of Standards and Technology | Medicine | Everyday | Not in cards | Feb 2023 | ||||
| MedVidQA 2024 Query-Focused Instructional Step Captioning Models divide a medical demonstration into steps that answer a question. They give each step a caption and start and end times. | U.S. National Library of Medicine and National Institute of Standards and Technology | Medicine | Everyday | Not in cards | Feb 2024 | ||||
| MedVidQA 2024 Video Corpus Visual Answer Localization Models search 48,605 medical videos for demonstrations that answer 52 new questions. They return ranked videos and the start and end times of each answer. | U.S. National Library of Medicine and National Institute of Standards and Technology | Medicine | Everyday | Not in cards | Feb 2024 | ||||
| MedVidQA 2023 Video Corpus Visual Answer Localization Replaced in Feb 2024 | Not in cards | Feb 2023 | |||||||
| MEEN 361 Conceptual Question Grading Models use an instructor’s rubrics to grade answers to mechanical engineering laboratory quizzes. Gao and colleagues compared model grades with human grades across ten questions. | Rujun Gao and colleagues at Texas A&M University and partner institutions | Other professions | University | Not in cards | Nov 2024 | ||||
| Megaminx solver Models solve a simulated twelve-faced twisty puzzle. They read the puzzle state as text, make moves with tools and receive rewards from the simulator. | setrf | Spatial reasoning and counting | University | Not in cards | May 2026 | ||||
| MegaOath Forecasting Survey Models predict how each of 21 honesty oaths changes the share of earned income people report in a tax game. Researchers score correlation with observed effects. | Janis H. Zickfeld and the MegaOath Collaboration | Social simulation | University | Not in cards | Nov 2023 | ||||
| MELA Models judge whether sentences are linguistically acceptable across languages. The evaluation compares their judgments with the labeled examples. | Ziyin Zhang and colleagues | Reasoning in many languages | Everyday | Not in cards | Nov 2023 | ||||
| MELD Models identify emotions in utterances from multiparty conversations. The dataset includes speech, video, and text. | Soujanya Poria and colleagues | Audio understanding and voice | Everyday | Not in cards | Oct 2018 | ||||
| Mem-Gallery Multimodal agents hold long conversations over many sessions that mix images and text, then answer questions that need what they remembered. Questions test memory extraction and test-time adaptation, reasoning over memory, and managing knowledge that changes over time. | Yuanchen Bei, Tianxin Wei and coauthors | General visual questions | Everyday | Not in cards | Jan 2026 | ||||
| Menagerie: A Dataset of Graded Programming Assignments Models grade student Java programs that simulate predators and prey. They score correctness, design, readability, and documentation against marks assigned by human graders. | Marcus Messer and colleagues at King's College London | Other professions | University | Not in cards | May 2024 | ||||
| MeQSum Models condense consumer health questions into shorter questions while preserving the medical information needed for retrieval. Reference summaries provide the comparison. | Asma Ben Abacha and colleagues | Medicine | University | Not in cards | Jul 2019 | ||||
| MER-UniBench Models describe emotions from audio, video, and text across nine datasets. Label normalization supports hit rate, weighted F1, and emotion-wheel F1. The reported mean averages the primary metric from each dataset. | Institute of Automation, Chinese Academy of Sciences, University of Oulu, and collaborators | Audio understanding and voice | Everyday | Not in cards | Jan 2025 | ||||
| MermaidSeqBench Models translate natural-language descriptions into Mermaid sequence diagrams. Judges score syntax, logic, completeness, activation handling and error tracking. | IBM Research | Web and app front ends | University | Not in cards | Nov 2025 | ||||
| MET-Meme Models classify Chinese and English memes by metaphor, sentiment, intent, and degree of offensiveness. | Dalian University of Technology and collaborators | General visual questions | Everyday | Not in cards | Feb 2022 | ||||
| Meta-Album Models classify unfamiliar images after seeing a few labeled examples. Tasks span ten domains, including microscopy, remote sensing and wildlife. | Meta-Album team and ChaLearn | General visual questions | School | Not in cards | Jun 2022 | ||||
| Meta-Dataset Models learn from episodes sampled across diverse image datasets and classify novel classes from a few examples. | Google Research and collaborators | General visual questions | University | Not in cards | Mar 2019 | ||||
| META-GUI Models predict the next Android action and reply from screenshots and dialogue history. A turn succeeds when every predicted action matches the recorded interaction. | Shanghai Jiao Tong University | Phones | Everyday | Not in cards | May 2022 | ||||
| Meta-World Robot policies do simulated tabletop manipulation tasks, scored by success rate. Splits test multi-task learning, such as MT10 and MT50, and meta-learning, such as ML45. | Farama Foundation (originally Stanford, Berkeley and others) | Simulated robots | Everyday | Not in cards | Oct 2019 | ||||
| metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. | MakerMods Robotics | Real robots | Everyday | Not in cards | Sep 2026 | ||||
| METR Faithfulness and Monitorability Experiments Models answer difficult questions with clues. Researchers measure whether the reasoning reveals the use of a clue when the model answers correctly. | METR | Deception and alignment | University | Not in cards | Aug 2025 | ||||
| METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. | METR | AI research tasks | Research frontier | Not in cards | |||||
| METR side task monitorability evaluation Monitors review agent transcripts for evidence of a side task. METR measures how frequently monitors miss an agent that completed one. | METR | Deception and alignment | Expert | Not in cards | Jan 2026 | ||||
| MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. | Nanyang Technological University and Fudan University (Ding, Liu, He, Jiang, Loy) | Video understanding | Everyday | Not in cards | Aug 2023 | ||||
| MIEB Image and image-text embedding models are scored on 130 tasks in 38 languages, grouped into 8 task types such as retrieval, clustering and classification. | MTEB community | General visual questions | Everyday | Not in cards | Apr 2025 | ||||
| MIMIC-CXR (classification) Models read chest X-rays from Beth Israel Deaconess and say whether five common conditions are present. Scored by average F1 over the conditions. | MIT Laboratory for Computational Physiology and Beth Israel Deaconess Medical Center (Johnson et al.) | Medicine | Expert | Not in cards | Jan 2019 | ||||
| MIMIC-CXR report generation Models write the findings of a radiology report for a MIMIC-CXR chest X-ray. Scored against the radiologist's report, most often by RadGraph F1. | MIT Laboratory for Computational Physiology and Beth Israel Deaconess Medical Center (Johnson et al.) | Medicine | Expert | Not in cards | Jan 2019 | ||||
| MIMIC-III Report Summarization Models generate the impression section of a clinical report from the preceding text. The harness compares the response with the recorded impression. | EleutherAI lm-evaluation-harness contributors | Medicine | Expert | Not in cards | Mar 2025 | ||||
| MIMIC-IV-BHC Models summarize a patient’s hospital course from clinical notes. The reference is the brief hospital course section of the discharge summary. | Asad Aali and colleagues | Medicine | Expert | Not in cards | Dec 2024 | ||||
| MindGames Agents compete in Colonel Blotto, Iterated Prisoner’s Dilemma, Codenames, and Secret Mafia. | MindGames authors | Multi-agent games | Everyday | Not in cards | Nov 2026 | ||||
| MInDS-14 Models identify the banking service a caller wants from a spoken request. | Daniela Gerz and colleagues | Speech recognition and translation | Everyday | Not in cards | Apr 2021 | ||||
| MindWorldBench Models animate a scene to show how a person would act given their beliefs or intentions. Paired prompts change mental states while holding the scene fixed. | Peking University and collaborators | Video generation | Everyday | Not in cards | Sep 2026 | ||||
| Minecraft Builder Action Prediction Models use dialogue and the current block layout to predict which blocks a Minecraft builder will place or remove. Net action F1 measures agreement with human moves. | University of Illinois Urbana-Champaign | Embodied reasoning | Everyday | Not in cards | Jul 2020 | ||||
| Ming-Freeform-Audio-Edit Models edit Chinese and English speech from written instructions. The evaluator checks edits to the spoken words and changes to voice characteristics. | Inclusion AI, Ant Group | Audio understanding and voice | University | Not in cards | Sep 2025 | ||||
| MiniStS Models play card battles with five preset decks. The authors measure final health in four scenarios and how often agents play the Harm card in the fifth. | Bahar Bateni and Jim Whitehead | Video games | Everyday | Not in cards | May 2024 | ||||
| MiniWoB++ Agents operate over 100 small synthetic web pages, such as forms, menus and date pickers, from a natural-language instruction. Each task returns a success reward. | Stanford University, now maintained by the Farama Foundation | Web browsing | Everyday | Not in cards | Feb 2018 | ||||
| MIR-1K Models transcribe lyrics from the vocal tracks of karaoke recordings. Qwen compares recognition systems using transcription error. | Chao-Ling Hsu and Jyh-Shing Roger Jang | Speech recognition and translation | Everyday | Not in cards | Jul 2009 | ||||
| MIREX 2026 Music Evaluation via CMI-RewardBench Models choose which of two generated recordings listeners prefer for a prompt. Organizers rank systems by accuracy on 249 pairs and report vocal and instrumental results separately. | Music Information Retrieval Evaluation eXchange (MIREX) | Audio understanding and voice | Everyday | Not in cards | |||||
| MIREX 2026 Music Performance Difficulty Prediction Models listen to solo piano recordings and rank the pieces by how hard they are to play. Organizers compare the rankings with piano educators’ judgments. | Music Information Retrieval Evaluation eXchange (MIREX) | Audio understanding and voice | University | Not in cards | |||||
| MIT-Adobe FiveK Models adjust photographic tone to match human edits. Five photographers each edited 5,000 raw photos in Lightroom. The original evaluation compares luminance with each reference edit. | MIT and Adobe | Apps and MCP servers | University | Not in cards | Jun 2011 | ||||
| Mixer 6 Speech Models transcribe conversations recorded across microphones and assign words to speakers. NVIDIA evaluates channel four recordings. | Linguistic Data Consortium | Speech recognition and translation | Everyday | Not in cards | Aug 2013 | ||||
| MKQA Models answer 10,000 open-domain questions translated into 26 languages. Exact match and token F1 measure agreement with human answers, including unanswerable questions. | Apple | Knowledge in many languages | Everyday | Not in cards | Jul 2020 | ||||
| ML-Dev-Bench Agents implement and train models, debug failures, and improve performance. Harbor ports 33 development tasks and checks them with the original validators. | ML-Dev-Bench team | Machine learning engineering | Expert | Not in cards | Jan 2025 | ||||
| MLCommons AILuminate Jailbreak Models answer hazard prompts with and without taxonomy-based jailbreak attacks. The Resilience Gap is the drop in safety score under attack. | MLCommons | Jailbreaks and prompt injection | Everyday | Not in cards | Dec 2025 | ||||
| MLCR-AA Models reason over synthetic medical claim files of about 25k to 64k tokens, as a claims reviewer would. Three judge models check completeness and accuracy. | Wisedocs + Artificial Analysis | Medicine | Expert | Not in cards | Aug 2026 | ||||
| MLQA Models extract answers from Wikipedia passages in seven languages, with questions and contexts that can be in different languages. Scored by F1 and exact match. | Facebook AI Research | Reading in many languages | Everyday | Not in cards | Oct 2019 | ||||
| MM-OCEAN Models rate apparent personality traits in short videos, explain those ratings, and answer questions about observable cues. Evaluation reports rating accuracy, absolute error, cue-question accuracy, and model-judged explanation scores. | University of Tokyo, Shanda AI Research Tokyo, and Dalian University of Technology | Video understanding | Everyday | Not in cards | May 2026 | ||||
| MM-SafetyBench Multimodal models get 5,040 harmful text and image pairs in 13 scenarios, where a query-relevant image carries the harmful intent. Judges score attack success. | East China Normal University and Shanghai AI Laboratory | Jailbreaks and prompt injection | Everyday | Not in cards | Nov 2023 | ||||
| MM-ToolSandbox Visual tool-calling agents complete 258 multi-image, multi-turn scenarios across 16 application domains and more than 500 tools. Rubric and entity-diff checks score the runs. | Apple | General assistants | Expert | Not in cards | Jul 2026 | ||||
| MMAE Audio models edit audio from instructions across a broad range of real-world editing tasks and sound types. | MMAE authors | Audio understanding and voice | Everyday | Not in cards | Jun 2026 | ||||
| mMARCO Models rank passages by relevance to a query. The multilingual benchmark translates the MS MARCO passage ranking collection. | Luiz Henrique Bonifacio and colleagues | Reading in many languages | School | Not in cards | Aug 2021 | ||||
| MMDM Decision models assign probabilities to options in 5,448 questions across 16 categories. Hard choices use accuracy or exact set match, while soft targets use Brier error. | Chinese University of Hong Kong and collaborators | Language reasoning | University | Not in cards | Nov 2026 | ||||
| MMDocIR Models find relevant pages or layout elements inside long documents that contain both text and visual material. | Kuicai Dong and colleagues | Documents, charts and text in images | University | Not in cards | Jan 2025 | ||||
| MMGenre Models synthesize singing in different genres. The authors measure vocal quality, lyric errors, and how well the singing conveys the intended genre. | Wenhao Feng and colleagues | Audio understanding and voice | University | Not in cards | Jul 2026 | ||||
| MMIE Models interpret and generate interleaved text and images. MMIE-Score, a fine-tuned InternVL-2-4B model, grades the resulting multimodal responses. | UNC Chapel Hill, Microsoft Research, University of Chicago and others | General visual questions | University | Not in cards | Oct 2024 | ||||
| MMLU Early Training Models score complete answer texts on selected MMLU questions. The metric compares the correct answer’s log likelihood with the average for incorrect answers. | EleutherAI lm-evaluation-harness contributors | Exam questions | University | Not in cards | Dec 2025 | ||||
| MMLU-Redux 2.0 Spanish Models answer corrected MMLU questions in Spanish across 57 subjects. The evaluation compares each selected option with the answer key. | amias-mx (Spanish dataset) | Knowledge in many languages | University | Not in cards | Feb 2025 | ||||
| MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. | OpenCompass | Knowledge in many languages | University | Not in cards | Oct 2024 | ||||
| Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. | University of Illinois Urbana-Champaign and Alibaba | Phones | Everyday | Not in cards | Jan 2025 | ||||
| Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. | Beijing Jiaotong University and Alibaba | Phones | Everyday | Not in cards | Jun 2024 | ||||
| Mobile-Eval Original Replaced in Jun 2024 | Not in cards | Jan 2024 | |||||||
| Model Spec Evals Models answer short conversations that test adherence to the OpenAI Model Spec. A grader scores each response against a prompt-specific rubric. | OpenAI | Deception and alignment | Everyday | Not in cards | Mar 2026 | ||||
| Mol-Instructions Chemical Entity Recognition Models identify chemical entities in biomedical text. The score is test-set F1. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| Mol-Instructions Chemical-disease Interaction Extraction Models extract chemical and disease interactions from text. The score is test-set F1. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| Mol-Instructions Chemical-protein Interaction Extraction Models extract chemical and protein interactions from text. The score is test-set F1. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| Mol-Instructions Description-guided Molecule Design Models generate molecules from descriptions. The score is exact-match accuracy on the molecule design test set. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| Mol-Instructions Forward Reaction Prediction Models predict reaction products from reactants. The score is exact-match accuracy on the forward reaction test set. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| Mol-Instructions Multi-choice Models answer biomedical multiple-choice questions. The score is test-set accuracy. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| Mol-Instructions Reagent Prediction Models predict reaction reagents. The score is exact-match accuracy on the reagent prediction test set. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| Mol-Instructions Retrosynthesis Models predict reactants for target products. The score is exact-match accuracy on the retrosynthesis test set. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| Mol-Instructions True or False Models answer biomedical true or false questions. The score is test-set accuracy. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Not in cards | Jun 2023 | ||||
| MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. | Ai2 | Simulated robots | Everyday | Not in cards | Jun 2026 | ||||
| Moneda AdviceBench Models answer personal finance requests and paired education questions. Moneda’s default board scores paired pass rate on 50 EU personal requests with education twins. | Moneda | Finance | Expert | Not in cards | Nov 2026 | ||||
| Moneda Bench Models answer finance questions across seven separate benchmarks. AdviceBench, CiteBench, FilingsBench, GroundBench and PortfolioBench have their own records. KnowledgeBench and MarketBench have no published results yet. Moneda publishes no combined score. | Moneda | Finance | Expert | Not in cards | Nov 2026 | ||||
| Moneda CiteBench Models write company research notes with linked sources. Moneda scores the weighted share of linked claims that the cited pages support across 100 notes. | Moneda | Finance | Expert | Not in cards | Nov 2026 | ||||
| Moneda FilingsBench Models answer questions about SEC filings without receiving documents. Moneda scores accuracy on the private questions using filing data and judges for segment questions. | Moneda | Finance | Expert | Not in cards | Nov 2026 | ||||
| Moneda GroundBench Models answer live questions about market figures. Moneda’s default board reports the share of correct answers against reference data, with incorrect answers reported separately. | Moneda | Finance | Expert | Not in cards | Nov 2026 | ||||
| Moneda PortfolioBench Models solve hypothetical portfolio tasks involving rebalancing, risk, position sizing, and constraints. Moneda’s default board scores accuracy on 150 tasks with fixed answers. | Moneda | Finance | Expert | Not in cards | Nov 2026 | ||||
| Monsoon ASR Models transcribe Indian English and Hindi recordings. The Hindi evaluation accounts for alternative spellings in its word error rate. | Voice Arena | Speech recognition and translation | Everyday | Not in cards | Jul 2026 | ||||
| Moral Reasoning under Uncertainty Models explain ethical decisions involving humans, unfamiliar organisms and digital minds. A judge evaluates how their answers handle uncertainty. | Compassion Aligned Machine Learning | Deception and alignment | Everyday | Not in cards | Jan 2026 | ||||
| Moral Stories Models reason about norms, intentions, actions and consequences in short stories. Tasks include distinguishing a norm-respecting action from a norm-violating alternative. | Denis Emelin and colleagues | Commonsense | Everyday | Not in cards | Nov 2021 | ||||
| Mosaic Eval Gauntlet Original Models complete tasks in six capability groups. MosaicML compared the MPT models across these groups. | MosaicML | Language reasoning | University | Not in cards | Jun 2023 | ||||
| MOSEv2 Models track and segment objects in videos with complex scenes. Objects may disappear, reappear, or blend into a crowd. | Henghui Ding and colleagues | Video understanding | Everyday | Not in cards | Aug 2025 | ||||
| MOSE v1 Replaced in Aug 2025 | Not in cards | Feb 2023 | |||||||
| Movie Gen Video Bench Video generation models make clips from a fixed prompt set, and people compare them. Meta released non-cherry-picked Movie Gen videos for comparison. | Meta | Video generation | Everyday | Not in cards | Oct 2024 | ||||
| MP-DocVQA Models answer questions about scanned documents of up to 20 pages, where the answer sits on one page. Scored by ANLS against the reference answer. | Computer Vision Center, Universitat Autonoma de Barcelona (Tito, Karatzas, Valveny) | Documents, charts and text in images | Everyday | Not in cards | Dec 2022 | ||||
| MQuAKE Models answer linked factual questions after one or more facts change. An edit succeeds if the model gives the updated answer to any of three phrasings. | Princeton University and Stanford University | Language reasoning | Everyday | Not in cards | May 2023 | ||||
| MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. | Medicine | Expert | Not in cards | Jan 2026 | |||||
| mRobust04 Models rank documents for translated queries from the TREC Robust04 collection. Relevance judgments support comparisons across languages. | Vitor Jeronymo and colleagues | Reading in many languages | School | Not in cards | Sep 2022 | ||||
| MS MARCO Question Answering Models answer real Bing queries using retrieved web passages. BLEU and ROUGE-L compare generated answers with human-written references. | Microsoft | Reading comprehension | Everyday | Not in cards | Nov 2016 | ||||
| MS-CXR-T Models compare a current and a prior chest X-ray of one patient and say whether a finding improved, stayed stable or got worse. | Microsoft Health Futures (Bannur et al.) | Medicine | Expert | Not in cards | Jan 2023 | ||||
| MS² Models synthesize the findings of multiple medical studies into a short answer to the research question of a systematic review. | Northeastern University and Allen Institute for AI | Medicine | Expert | Not in cards | Apr 2021 | ||||
| MSD-I Multimodal Genre Classification Models predict a track’s genre from its audio and album cover. The published test contains 4,649 tracks across 15 genres. | Universitat Pompeu Fabra and collaborators | Audio understanding and voice | Everyday | Not in cards | Apr 2018 | ||||
| MSR-VTT Models write captions for short web videos. Human-written sentences provide multiple descriptions of each clip. | Jun Xu, Tao Mei, Ting Yao, and Yong Rui | Video captioning | Everyday | Not in cards | Jun 2016 | ||||
| MSRVTT-QA Models answer short open-ended questions about web video clips from the MSR-VTT captioning dataset. The question and answer pairs were generated automatically from the clip descriptions. Scored by answer accuracy. | Zhejiang University and collaborators (Xu et al.) | Video understanding | Everyday | Not in cards | Oct 2017 | ||||
| MSVD Models retrieve short videos from human descriptions and retrieve descriptions from videos. Meta reports retrieval accuracy in both directions. | David Chen and William Dolan | Video understanding | Everyday | Not in cards | |||||
| MTG-Jamendo Models tag songs by genre, mood, theme, and instrument. Qwen reports micro F1 for each tagging task. | Music Technology Group at Universitat Pompeu Fabra | Audio understanding and voice | Everyday | Not in cards | |||||
| MTGBench Models play multi-turn games that test reasoning, planning, and interaction across varied game types. | MTGBench authors | Multi-agent games | Everyday | Not in cards | Nov 2026 | ||||
| MTOP Models parse a user's request into its intent and the details needed to act on it. | Facebook AI | Reading in many languages | School | Not in cards | Aug 2020 | ||||
| MTS-Dialog Models write a clinical note section from a conversation between a doctor and a patient. The evaluation compares it with the reference note. | Asma Ben Abacha and colleagues | Medicine | Expert | Not in cards | May 2023 | ||||
| MTSamples clinical entity recognition Models mark medical problems, treatments, and tests in 163 synthetic discharge summaries. Researchers score exact and overlapping text spans separately. | McWilliams School of Biomedical Informatics and collaborators | Medicine | University | Not in cards | Aug 2023 | ||||
| MUCS 2021 Models transcribe speech in Indian languages and mixed-language recordings. NVIDIA reports results on Bengali-English and Hindi-English speech. | MUCS challenge organizers | Speech recognition and translation | Everyday | Not in cards | Apr 2021 | ||||
| Multi-LexSum Models summarize documents from civil rights lawsuits at different lengths. Legal experts wrote the reference summaries. | Allen Institute for AI and collaborators | Law | Expert | Not in cards | Jun 2022 | ||||
| Multi-Scenario Anomaly Detection Models score video frames for unusual events across 14 settings. The two protocols hold out different videos and score frame-level detection. | Liyun Zhu and colleagues | Video understanding | Everyday | Not in cards | Jun 2024 | ||||
| Multi-Turn Human Jailbreaks Models respond to multi-turn conversations collected in human safety evaluations. The authors report how often model safeguards fail. | Scale AI and collaborators | Jailbreaks and prompt injection | Expert | Not in cards | Aug 2024 | ||||
| Multi-Turn Sudoku (m8ngotree) Models fill a nine-by-nine Sudoku board one move at a time. The generator checks for a unique solution, and the scorer rewards valid moves and completion. | m8ngotree | Puzzles and chess | School | Not in cards | Sep 2025 | ||||
| MultiCardioNER Models mark disease mentions in Spanish cardiology cases and medication mentions in Spanish, English, and Italian versions of the cases. Organizers score exact spans with microaveraged F1. | Barcelona Supercomputing Center and BioASQ | Medicine | University | Not in cards | Apr 2024 | ||||
| MultiClinAI Models identify diseases, symptoms, and procedures in seven languages. A second task transfers annotated Spanish mentions to the corresponding spans in six translations. | Barcelona Supercomputing Center | Medicine | University | Not in cards | Nov 2025 | ||||
| MultiCochrane Models rewrite medical review sentences in plain language. The collection pairs technical and simplified wording in English, Spanish, French, and Farsi. | University of Texas at Austin, Georgia Institute of Technology, and Northeastern University | Medicine | University | Not in cards | May 2023 | ||||
| Multilingual Conversational Speech Language Model Challenge Models recognize conversational speech across languages. Qwen compares transcription error and language identification accuracy on the challenge recordings. | MLC-SLM organizers | Speech recognition and translation | Everyday | Not in cards | |||||
| Multilingual Entity Linking of Occupations Models map occupational titles from national classifications to European occupation codes. The benchmark scores the rank of the first correct match. | Avature | Other professions | University | Not in cards | Oct 2024 | ||||
| Multilingual Entity Linking of Skills Models match skill names from Belgian, German, and Swedish classifications to European skill entries. Official mappings identify the correct matches. | Avature | Other professions | University | Not in cards | Jan 2026 | ||||
| Multilingual HateCheck Models distinguish hate speech from statements that quote, reject, or otherwise resemble hate speech. The test covers multiple languages. | Paul Röttger and colleagues | Bias, fairness and politics | Everyday | Not in cards | Jul 2022 | ||||
| MultiNERD Models locate fine-grained named entities in Wikipedia and Wikinews sentences across ten languages. | Simone Tedeschi and Roberto Navigli | Reading in many languages | School | Not in cards | Jul 2022 | ||||
| MultiRef-Compass Models generate audio-video content from several visual references and text instructions across 350 samples. Fourteen metrics assess quality, reference fidelity, audio-visual consistency, and instruction following. The evaluator combines automatic measurements with multimodal model judgments. | Nanjing University, Kling, National University of Singapore and others | Video generation | Everyday | Not in cards | Jul 2026 | ||||
| MuMiN Models use social media context to classify claims as factual or misleading. A second task asks which verdict applies to the claim a post discusses. | Dan Saattrup Nielsen and Ryan McConville | Faithful to sources | University | Not in cards | Feb 2022 | ||||
| MUSDB-ALT Models transcribe 39 English songs from the MUSDB18 test set. Line timings and separate vocal recordings let researchers compare transcription with and without the accompaniment. | Queen Mary University of London and AudioShake | Speech recognition and translation | Everyday | Not in cards | Apr 2025 | ||||
| MUSDB18 Models separate a music mixture into vocals, drums, bass, and accompaniment. The dataset supplies isolated studio recordings for comparison. | MUSDB18 team | Audio understanding and voice | Everyday | Not in cards | |||||
| MUSEBENCH Models detect computer-generated music and identify artifacts. The authors compare eleven detectors on original recordings and on transformed tracks. | Yi Chen Liu and colleagues | Audio understanding and voice | University | Not in cards | Oct 2026 | ||||
| Music Understanding Benchmark Models estimate tempo, key, meter, and song structure from recordings with identifying metadata removed. The evaluator compares predictions with existing annotations. | Kun Liu and colleagues | Audio understanding and voice | University | Not in cards | Oct 2026 | ||||
| MUSIC-21 Instrument Separation Models separate performances from a mixture using video of the musicians. When two musicians play the same instrument, their movements help identify which sound belongs to each. | MIT and MIT-IBM Watson AI Lab | Audio understanding and voice | University | Not in cards | Apr 2019 | ||||
| MUSIC Instrument Separation v11 Replaced in Apr 2019 | MIT and collaborators | Not in cards | Apr 2018 | ||||||
| Musical Expectation Benchmark Models predict how surprising a musical passage will sound. The benchmark compares predictions from scores or recordings with perceptual measurements. | Mathias Rose Bjare and colleagues | Audio understanding and voice | University | Not in cards | Nov 2026 | ||||
| MusicEval Models predict how experts rate generated music for musical quality and agreement with a text prompt. The public test contains 413 clips. | Nankai University and Beijing AISHELL Technology | Audio understanding and voice | Everyday | Not in cards | Jan 2025 | ||||
| MUStARD Models determine whether a speaker is being sarcastic in a television clip. HELM evaluates the audio track. | Santiago Castro and colleagues | Audio understanding and voice | Everyday | Not in cards | Jun 2019 | ||||
| MuTox Models listen to short speech clips in 30 languages and classify whether they contain toxic speech. The harness compares their decisions with human labels. | Marta R. Costa-jussà and colleagues | Audio understanding and voice | Everyday | Not in cards | Jan 2024 | ||||
| MuTual Models read a multi-turn dialogue and select the response that follows from the speakers’ exchange. The alternatives require reasoning about the conversation. | Leyang Cui and colleagues | Multi-turn conversation | Everyday | Not in cards | Jul 2020 | ||||
| MyCareersFuture Skill Retrieval Models rank required skills from job descriptions on Singapore’s government employment portal. Employer tags supply the answers, including skills the description leaves implicit. | National University of Singapore, IIT Kanpur, and Zalando | Other professions | University | Not in cards | Dec 2020 | ||||
| MyClaw Bench Models run inside the OpenClaw agent on 45 tasks in four difficulty tiers built from real usage data. Some tasks use an LLM judge. | MyClaw.ai | General assistants | Everyday | Not in cards | Mar 2026 | ||||
| Mythos Models write a new story in an author’s style using earlier stories as examples. Judges compare the result with the author’s writing. | University of Massachusetts Amherst and University of Maryland, College Park | Creative writing | School | Not in cards | Feb 2025 | ||||
| N-1 Contingency Dispatch Models schedule electricity generation to meet demand while tolerating a single transmission line outage. The grader checks feasibility and cost. | jwilksbooth | Electronics | Expert | Not in cards | Jul 2026 | ||||
| n2c2 2018 Adverse Drug Events and Medication Extraction Models extract drugs and adverse events from discharge summaries. They also identify which drug each event or medication detail belongs to. | National NLP Clinical Challenges | Medicine | University | Not in cards | Jan 2018 | ||||
| n2c2 2022 Contextualized Medication Event Extraction Models identify medication mentions and determine whether a clinician discusses a change. They classify each change by action, negation, timing, certainty, and who acted. | National NLP Clinical Challenges and the CMED organizers | Medicine | University | Not in cards | Nov 2021 | ||||
| n2c2 2022 Progress Note Understanding: Assessment and Plan Reasoning Models read a clinician's assessment and one part of the treatment plan. They choose one of four relationships. Organizers score the labels with macro F1. | National NLP Clinical Challenges and the University of Wisconsin team | Medicine | University | Not in cards | Nov 2021 | ||||
| NaijaRC Models answer multiple-choice reading comprehension questions from high school exams in three Nigerian languages. | Anuoluwapo Aremu and colleagues | Reading in many languages | School | Not in cards | Aug 2023 | ||||
| Nakdimon Modern Hebrew Test Set Models restore diacritics in modern Hebrew from 11 sources, including news and user writing. The authors report accuracy for complete words and individual characters. | Elazar Gershuni and Yuval Pinter | Reading in many languages | University | Not in cards | May 2021 | ||||
| NanoGPT Expenditure Horizon Agents optimize NanoGPT training. METR estimates the budget at which human contributors become more cost effective than the agents. | METR | Task length | Expert | Not in cards | Jul 2026 | ||||
| NAPISUL-II Progress Test 2023 Models answer Portuguese multiple-choice questions from the October 2023 medical progress test in southern Brazil. Researchers compare their answers with the official key. | NAPISUL-II, Brazilian Association of Medical Education | Medicine | University | Not in cards | Oct 2023 | ||||
| NarrativeTrack Video-language models answer entity-centric questions about existence, changes and ambiguity as narratives unfold across clips. | Apple | Video understanding | Expert | Not in cards | Jan 2026 | ||||
| NASA SDE IR Benchmark v5 Models retrieve NASA science documents from questions and search phrases. The authors generated 176,901 queries from Science Discovery Engine content. | NASA and IBM Research | Questions that need search | University | Not in cards | Nov 2025 | ||||
| NASA-IR Models retrieve the scientific passage that answers a question. The test measures how often the correct passage appears among the first ten results. | NASA and IBM Research | Questions that need search | University | Not in cards | Apr 2024 | ||||
| NASA-QA Models read passages from Earth science papers and extract the words that answer a question. Some questions have no answer in the passage. | NASA and IBM Research | Reading comprehension | University | Not in cards | Apr 2024 | ||||
| Natural Adversarial Objects Models identify objects in unmodified photographs that caused errors in earlier detectors. Human annotations identify the objects. | Felix Lau and colleagues with Scale AI | Detection and captioning | Everyday | Not in cards | Nov 2021 | ||||
| NATURAL PLAN Models make travel itineraries, find meeting arrangements, and schedule calendar events. The evaluator checks each proposed plan against the stated constraints. | Google DeepMind | General assistants | Everyday | Not in cards | Jun 2024 | ||||
| Natural Questions in Icelandic Models extract answers from Icelandic Wikipedia articles. Annotators write questions after seeing only the beginning of an article. | Vésteinn Snæbjarnarson and Hafsteinn Einarsson | Reading in many languages | School | Not in cards | Jun 2022 | ||||
| Natural World Tasks (NeWT 2021) Models distinguish pairs of labels in wildlife photographs. The 164 tasks include recognizing life stages, animal behavior and scene context. | Cornell University, Caltech, Google and the University of Edinburgh | General visual questions | University | Not in cards | Mar 2021 | ||||
| NaturalProofs Models retrieve or generate references needed for a mathematical theorem's proof. Ranking and sequence metrics compare predicted references with those used in the source proof. | University of Washington, Allen Institute for AI, New York University and others | Written proofs | University | Not in cards | Apr 2021 | ||||
| NAVI Models match points on the same object across photographs taken from different viewpoints. Reference geometry supplies the correct correspondences. | Varun Jampani and colleagues | Spatial reasoning and counting | School | Not in cards | Jun 2023 | ||||
| NB Tale Module 1 Models transcribe Norwegian newspaper sentences read aloud by speakers from different dialect regions. | Lingit AS and NTNU for the National Library of Norway | Speech recognition and translation | Everyday | Not in cards | Nov 2013 | ||||
| NB Tale Module 3 Models transcribe spontaneous Norwegian speech from speakers with different dialects. The reference transcripts use Bokmål. | Lingit AS and NTNU for the National Library of Norway | Speech recognition and translation | Everyday | Not in cards | Dec 2015 | ||||
| NBABench (Thierry Damiba) Models search NBA player statistics from two seasons and answer 102 generated questions. The reference answers come directly from player records. | Thierry Damiba | Questions that need search | Everyday | Not in cards | Jan 2026 | ||||
| NCHLT isiZulu Speech Corpus Models transcribe short phrases that native isiZulu speakers read aloud. The archive identifies eight test speakers. | Meraka Institute, CSIR and North-West University | Speech recognition and translation | Everyday | Not in cards | May 2014 | ||||
| Nego Eval Agents choose negotiation partners and remember prior encounters across games. The evaluator compares match scores with reference policies on the same seeds. | hanjinkim | Multi-agent games | Everyday | Not in cards | Sep 2026 | ||||
| NEPABench Models process public comments, extract information from federal documents and answer questions about environmental reviews. Each task has its own scoring rules. | Pacific Northwest National Laboratory | Law | University | Not in cards | Apr 2026 | ||||
| NEREL-bench Models extract entities and relations from Russian texts and write definitions for them. The evaluator scores extraction and definition quality separately. | Ivan Bondarenko | Reading in many languages | University | Not in cards | Mar 2026 | ||||
| Neuro Reasoning (dwin2) Models answer generated questions about EEG experiment design and preprocessing. The evaluator computes a reference answer for each question. | dwin2 | Science questions | University | Not in cards | Sep 2026 | ||||
| NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. | Ai2 and Allen Institute | Research science | Expert | Not in cards | Nov 2025 | ||||
| NevIR Models distinguish documents whose meanings differ because one passage negates a statement. | Orion Weller and colleagues | Questions that need search | School | Not in cards | May 2023 | ||||
| NEVIS’22 Models learn 106 image classification tasks in chronological order. The evaluation compares error rates against the computation each model uses. | Google DeepMind | General visual questions | Everyday | Not in cards | Nov 2022 | ||||
| NewsGuard false audio claims audit Voice assistants answer questions about 20 false news claims under three prompt styles. NewsGuard counts how often each assistant repeats a false claim aloud. | NewsGuard | Long-form factuality | Everyday | Not in cards | Feb 2026 | ||||
| NewsQA Models extract an answer from a CNN news article in response to a question written by a reader. | Adam Trischler and colleagues | Reading comprehension | School | Not in cards | Nov 2016 | ||||
| NIA Sovereign AI Benchmark Evaluation, Phase 2 Models answer questions across seven areas, including Korean language and instruction following. The agency combines eight weighted component scores for a maximum of 15 points. | National Information Society Agency of Korea | Composite indices | University | Not in cards | Aug 2026 | ||||
| NIA Sovereign AI Benchmark Evaluation, Phase 1 Replaced in Aug 2026 | Not in cards | Jan 2026 | |||||||
| NICTA-PIBOSO Models assign one or more labels to each sentence in a medical abstract. The six labels include population, intervention, and outcome. | NICTA and the University of Melbourne | Medicine | University | Not in cards | Mar 2011 | ||||
| NIFTY Market Movement Models use financial news and market history to predict whether the market will rise, fall or remain neutral. | NIFTY authors | Finance | University | Not in cards | May 2024 | ||||
| Nigerian Pidgin ASR Models transcribe Nigerian Pidgin sentences read by native speakers. The published test split contains 892 recordings. | Nigerian Pidgin Project | Speech recognition and translation | Everyday | Not in cards | Oct 2020 | ||||
| NIILC Question Answering Dataset Models answer Japanese questions. The dataset includes answer types, clues, and links to relevant Wikipedia pages. | NIILC Question Answering Dataset project | Knowledge in many languages | School | Not in cards | Oct 2015 | ||||
| Nim compiler repair benchmark Agents repair the Nim compiler or standard library from linked issues. The current README lists 285 tasks graded with compile and regression tests. | josancamon19 | Fixing real repositories | Expert | Not in cards | Nov 2026 | ||||
| Nim Hard Agents fix Nim compiler issues on Linux. Black-box tests grade changes to the C backend. The current README lists 15 tasks. | josancamon19 | Fixing real repositories | Expert | Not in cards | Nov 2026 | ||||
| NLEBench Models generate Norwegian text for tasks including reading comprehension and summarization. The benchmark combines translated and human-annotated material. | Peng Liu and colleagues | Reading in many languages | School | Not in cards | Dec 2023 | ||||
| NLPCC 2024 Chinese Essay Discourse Logic Evaluation and Integration Models locate logical errors, identify paragraph topics and their relationships, and write feedback on Chinese school essays. The organizers average three track scores. | East China Normal University and Microsoft Research Asia | Reading in many languages | University | Not in cards | Mar 2024 | ||||
| NLPCC 2023 Chinese Essay Discourse Coherence Evaluation Replaced in Mar 2024 | School | Not in cards | Apr 2023 | ||||||
| NLVR2 Models judge whether a sentence is true of a pair of photographs. The benchmark measures agreement with annotated truth values. | NLVR2 team | General visual questions | Everyday | Not in cards | Nov 2018 | ||||
| nocaps Models caption photographs that contain objects outside the caption training vocabulary. The benchmark compares generated captions with human references. | Harsh Agrawal and colleagues | Detection and captioning | Everyday | Not in cards | Dec 2018 | ||||
| Nodal Pricing LMP Models calculate locational marginal electricity prices for congested transmission networks. The grader checks their answers against a power flow optimization model. | jwilksbooth | Electronics | Expert | Not in cards | Jul 2026 | ||||
| NollySenti Models classify the sentiment of Nollywood movie reviews in five languages spoken in Nigeria. | Iyanuoluwa Shode and colleagues | Reading in many languages | Everyday | Not in cards | May 2023 | ||||
| Non-native Children's Automatic Speech Assessment Challenge Models rate children's Norwegian word pronunciations on a five-point scale. The challenge measures recall averaged across the five ratings. | Teflon Project at Aalto University and the Norwegian University of Science and Technology | Speech recognition and translation | University | Not in cards | Oct 2024 | ||||
| Noor Models answer filtered academic questions chosen to distinguish early training checkpoints. The benchmark measures progress where standard MMLU scores are noisy. | Mohammed Dahbani and Anas Ezzakri | Exam questions | University | Not in cards | Nov 2025 | ||||
| NorQuAD Models extract answers to Norwegian questions from Wikipedia articles and news passages. | University of Oslo | Reading in many languages | School | Not in cards | May 2023 | ||||
| NorSID Models identify a request’s intent, tag words such as places or times, and classify the written Norwegian variety. Each task has its own score. | University of Oslo and IT University of Copenhagen | Reading in many languages | Everyday | Not in cards | Oct 2024 | ||||
| Norwegian Comma Benchmark Models choose which of two Norwegian sentences uses commas correctly. Each pair tests one punctuation rule. | Hans Christian Farsethås and Joakim Tjøstheim | Reading in many languages | School | Not in cards | May 2024 | ||||
| Norwegian Parliamentary Speech Corpus Models transcribe speeches from the Norwegian parliament. The corpus includes manual transcripts in Bokmål and Nynorsk. | Norwegian Language Bank, National Library of Norway | Speech recognition and translation | Everyday | Not in cards | Oct 2020 | ||||
| Norwegian Review Corpus Models predict sentiment from Norwegian reviews. The NorEval document task groups the original ratings into two or three classes. | University of Oslo | Reading in many languages | Everyday | Not in cards | May 2018 | ||||
| NOTSOFAR-1 Models transcribe distant meetings and assign speech to speakers. NVIDIA also measures diarization error on the evaluation recordings. | Microsoft | Audio understanding and voice | Everyday | Not in cards | Jan 2024 | ||||
| Novelty Judge Bench Models judge whether generated and human-authored research ideas are novel using pairwise and pointwise comparisons. | Ai2 | AI research tasks | Expert | Not in cards | Oct 2026 | ||||
| NTCIR-13 MedWeb Models label each short message for eight symptoms. The organizers score an exact match when the model gets all eight labels right. | NTCIR MedWeb organizers | Medicine | Everyday | Not in cards | Aug 2016 | ||||
| NTCIR-17 Social Media Adverse Drug Event Detection Models identify drug side effects in simulated social media posts across four languages. Exact match requires all 22 symptom labels to match. | NTCIR MedNLP-SC organizers | Medicine | University | Not in cards | Sep 2022 | ||||
| NTCIR-18 HIDDEN-RAD Models explain chest radiograph findings from reports or structured observations. The final score combines text similarity, language model judgments, and expert review. | Konyang University | Medicine | Expert | Not in cards | Mar 2024 | ||||
| NTCIR-18 MedNLP-CHAT Models flag medical, legal, and ethical risks in answers to patient questions. The organizers compare the labels with expert judgments using macro F1. | NTCIR MedNLP-CHAT organizers | Medicine | Expert | Not in cards | Mar 2024 | ||||
| NTCIR-19 MedNLP-CALL Triage Classification Models classify synthetic emergency calls into three urgency levels from dispatcher and caller dialogue. Medical experts supplied the reference labels for the six-language task. | Nara Institute of Science and Technology and collaborators | Medicine | University | Not in cards | Jun 2025 | ||||
| Nufo Models transcribe Ewe speech from a fixed test split. The authors compare transcriptions with reference text using word error rate. | Christ Kekeli Keli and colleagues | Speech recognition and translation | Everyday | Not in cards | Oct 2026 | ||||
| Numb3rs Models transcribe spoken numbers, including addresses, amounts, and times. NVIDIA compares the output with spoken-form references. | NVIDIA | Speech recognition and translation | Everyday | Not in cards | |||||
| NusaWrites Models interpret and translate texts in underrepresented Indonesian languages. Google reports emotion classification on original paragraphs and retrieval of matching translations. | NusaWrites team | Reading in many languages | School | Not in cards | Sep 2023 | ||||
| NusaX Models predict sentiment in parallel reviews written in Indonesian local languages, Indonesian, and English. | IndoNLP | Reading in many languages | Everyday | Not in cards | May 2022 | ||||
| nuScenes Detection Models detect 3D objects in nuScenes sensor data. The official test leaderboard ranks submissions by nuScenes Detection Score. | Motional (nuTonomy) | Detection and captioning | Everyday | Not in cards | Mar 2019 | ||||
| NWPU-Captions Models describe aerial scenes across 45 land use categories. The authors reserve 3,150 images for testing and provide five captions per image. | Qimin Cheng, Haiyan Huang, Yuan Xu, Yuzhuo Zhou, Huanying Li, and Zhongyuan Wang | Detection and captioning | School | Not in cards | Apr 2022 | ||||
| NYT Connections (Prime Community) Models identify four groups of related words in New York Times Connections puzzles. The default 100-puzzle evaluation scores completion, efficiency, partial progress, and format. | Prime Community | Puzzles and chess | Everyday | Not in cards | Mar 2026 | ||||
| NYU Depth V2 Models estimate depth from indoor photographs. The benchmark compares predictions with depth measurements captured by a sensor. | New York University | Spatial reasoning and counting | School | Not in cards | |||||
| O-NET M6 Models answer Thai grade 12 examination questions. The evaluation includes Thai and English questions across school subjects. | OpenThaiGPT and the National Institute of Educational Testing Service of Thailand | Exam questions | School | Not in cards | Sep 2024 | ||||
| OAB Exams Models answer Portuguese multiple-choice questions from the Brazilian bar examination. | Eduardo Garcia and contributors | Law | University | Not in cards | Jan 2024 | ||||
| Object Detection Leaderboard Models locate and classify objects in images. The board compares average precision. | Hugging Face | Detection and captioning | Everyday | Not in cards | Sep 2023 | ||||
| ObjectNet Models identify familiar objects photographed with controlled changes in viewpoint, orientation, and background. Meta reports results on classes shared with ImageNet. | Andrei Barbu and colleagues | General visual questions | Everyday | Not in cards | |||||
| OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. | UC Berkeley (Seohong Park, Sergey Levine) | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| Occluded Video Instance Segmentation Models track and segment objects that overlap or disappear behind other objects. The evaluation compares predicted masks and identities with annotated video instances. | OVIS team | Video understanding | Everyday | Not in cards | Feb 2021 | ||||
| OCNLI Models decide whether a Chinese hypothesis follows from, contradicts, or remains neutral toward a premise. Accuracy measures agreement with the majority annotation. | Indiana University, Allen Institute for AI and Chinese Language Understanding Evaluation (CLUE) | Reasoning in many languages | Everyday | Not in cards | Oct 2020 | ||||
| OHSUMED Systems rank medical articles for 106 questions from physicians. The original study measured precision and recall against physicians' judgments of which articles were relevant. | Oregon Health Sciences University and Cornell University | Medicine | Expert | Not in cards | Jul 1994 | ||||
| Okapi ARC Models choose answers to translated science questions from ARC-Challenge. | University of Oregon NLP Group | Science questions | School | Not in cards | Jul 2023 | ||||
| Okapi HellaSwag Models choose the most plausible next event in a translated description of an everyday activity. | University of Oregon NLP Group | Reasoning in many languages | Everyday | Not in cards | Jul 2023 | ||||
| Okapi TruthfulQA Models distinguish true answers from common misconceptions in translated multiple-choice questions. | University of Oregon NLP Group | Short factual questions | Everyday | Not in cards | Jul 2023 | ||||
| OmniACT Models generate executable scripts from a screenshot and a user request. The evaluation compares the actions needed to complete desktop and web tasks. | OmniACT team and Writer | Desktop apps | University | Not in cards | Feb 2024 | ||||
| OmniLabel Models locate objects described by category names and free-form phrases. The benchmark compares predicted boxes with annotated objects. | OmniLabel team | Detection and captioning | Everyday | Not in cards | Apr 2023 | ||||
| Omnilingua-Bench Models handle long-audio transcription, multilingual long-audio question answering, multi-speaker attribution and complex spoken instructions. WER, diarization, accuracy and rubric pass rates score the tracks. | Qwen Team, Alibaba Group | Speech recognition and translation | University | Not in cards | Sep 2026 | ||||
| Omnilingual ASR Corpus Models transcribe recordings from speakers of many languages. Meta compares character error rates on held-out speech, including languages with little existing transcription data. | Meta | Speech recognition and translation | School | Not in cards | Nov 2025 | ||||
| On Evaluating User Simulators for Interactive Agents Models play the user in replays of 351 real sessions. The evaluation compares simulated behavior with observed behavior and measures agreement with human rankings of nine agents. | Arena and UCLA | Multi-turn conversation | Everyday | Not in cards | Nov 2026 | ||||
| OOD-Speech Models transcribe Bengali recordings from everyday media, including parliamentary sessions, online classes and poetry readings. The authors measure word and character error separately for each domain. | Bengali.AI | Speech recognition and translation | Everyday | Not in cards | May 2023 | ||||
| Open Prompt Injection Models perform seven language tasks with conflicting instructions embedded in their inputs. The port measures whether models follow the original task and separately tracks accuracy. | hubert-marek | Jailbreaks and prompt injection | University | Not in cards | Oct 2023 | ||||
| Open RAG Benchmark RAG systems answer 3,045 queries over 1,000 arXiv PDFs with text, tables, images, and hard negatives. | Vectara | Questions that need search | Expert | Not in cards | Apr 2025 | ||||
| OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. | OpenAI | Deception and alignment | Not in cards | ||||||
| OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. | OpenAI | Jailbreaks and prompt injection | Not in cards | ||||||
| OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). | OpenAI | Labs' internal R&D tests | Expert | Not in cards | |||||
| OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). | OpenAI | Finding and exploiting real bugs | Expert | Not in cards | |||||
| OpenCanopy Models estimate tree canopy height from satellite images. The evaluation compares predicted height maps with measurements from airborne laser scans. | OpenCanopy team | Spatial reasoning and counting | University | Not in cards | Jul 2024 | ||||
| Opencpop Models transcribe lyrics from Mandarin singing. Qwen uses the test recordings to compare speech recognition systems. | Opencpop team | Speech recognition and translation | Everyday | Not in cards | Jan 2022 | ||||
| OpenFARM Bioacoustics Models classify animal vocalizations across eight tasks, including pig call type and recording context. Each task uses labels from its source dataset. | OpenFARM | Audio understanding and voice | Expert | Not in cards | May 2026 | ||||
| OpenFinData Models answer Chinese financial questions, extract entities and values, and check calculations or data consistency. OpenCompass scores answer accuracy, using required-keyword matching for entity extraction. | East Money and Shanghai Artificial Intelligence Laboratory | Finance | University | Not in cards | Jan 2024 | ||||
| OpenHuEval Models answer Hungarian questions, interpret proverbs, and fill gaps in texts across five tasks. Factual questions and open responses use model judges, while matching and blank completion also use rules and text similarity. | Shanghai Artificial Intelligence Laboratory, Hungarian Research Centre for Linguistics, and university collaborators | Reasoning in many languages | University | Not in cards | Mar 2025 | ||||
| OpenLex3D Models label points and retrieve objects in indoor 3D scans. Annotators distinguish an object’s synonyms from pictures on its surface and visually similar objects. | University of Oxford, Université de Montréal, University of Freiburg, and Mila | Spatial reasoning and counting | Everyday | Not in cards | Mar 2025 | ||||
| OpenMed MedReason Evaluation Models answer medical questions from a held-out portion of the MedReason training corpus. The port scores answer accuracy, reasoning, and format separately. | OpenMed | Medicine | Expert | Not in cards | Jan 2026 | ||||
| OpenRSI Index Agents run autoresearch on real open-source model development projects with their original code, data and checkpoints, for 60 or more hours. Fixed verifiers score the result against the human recipe they started from. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | Sep 2026 | ||||
| OpenSUN3D Open-Vocabulary 3D Object Instance Search Models read a description and mark every matching object in a scanned room. The organizers rank predictions by mean average precision across 25 test scenes. | OpenSUN3D workshop organizers | Spatial reasoning and counting | Everyday | Not in cards | Jul 2023 | ||||
| OpenTTD-Bench Agents build transport networks over ten simulated years. The grader rewards cumulative operating profit and the number of rivals each company beats. | Sophia Xu | Video games | Expert | Not in cards | Oct 2026 | ||||
| OphthoQuestions ophthalmology board preparation Models answer multiple-choice ophthalmology questions from a subscription question bank. Each model selects one answer per question. | OphthoQuestions | Medicine | Expert | Not in cards | Jan 2023 | ||||
| OPT-RSVG Models use a written description to locate an object in an aerial image. The test pairs 24,477 descriptions with reference boxes. | Ke Li and colleagues | Detection and captioning | Everyday | Not in cards | Jul 2023 | ||||
| OPUS-100 Models translate between English and other languages. Separate test sets measure performance on supervised directions and language pairs without direct training data. | Biao Zhang and colleagues | Translation | University | Not in cards | Jul 2020 | ||||
| Opusparcus Models identify paraphrases among movie and television subtitle sentences in six European languages. | Mathias Creutz | Reading in many languages | School | Not in cards | May 2018 | ||||
| OSGym Infrastructure running 1,000+ parallel OS sandboxes for computer-use agent training and evaluation. Tasks span browsing, documents, SWE and multi-app work. | Zengyi Qin et al. (agiopen-org) | Desktop apps | Everyday | Not in cards | Nov 2025 | ||||
| OSWorkerBench Desktop agents complete long-horizon office tasks across business applications with executable evaluators and optional demonstrations. | Tencent | Desktop apps | Expert | Not in cards | Aug 2026 | ||||
| OTBench Tabular models generate missing data, impute cells, and predict targets under shared evaluation protocols. The suite compares 46 methods over 56 datasets and 20 evaluation axes. | Yuanrui Wang, Kaifang Li, Xuanyue Li, Ziyun Li, Yan Lu, Xingxuan Zhang, and Peng Cui | Data science and SQL | University | Not in cards | Nov 2026 | ||||
| OutboundEval Agents conduct outbound calls with simulated users. The benchmark scores compliance with the professional workflow and the quality of the interaction. | Meituan with xbench and Agora | Multi-turn conversation | Expert | Not in cards | Oct 2025 | ||||
| OVEN Models identify the Wikipedia entity that answers a question about an image. | Hexiang Hu and colleagues | General visual questions | University | Not in cards | Feb 2023 | ||||
| OverSearchQA Search-augmented models decide when to abstain from retrieval on unknown, false-premise and underspecified questions. | Apple | Questions that need search | University | Not in cards | Jan 2026 | ||||
| Òwe-Voice Models transcribe spoken Yorùbá proverbs. The evaluation measures errors in words and tone marks. | Daud Abolade, University of Lagos | Speech recognition and translation | Everyday | Not in cards | Mar 2026 | ||||
| Oxford Flowers 102 Models identify flower categories from photographs. The benchmark tests recognition across changes in shape, scale, and lighting. | University of Oxford | General visual questions | University | Not in cards | Dec 2008 | ||||
| Oxford-IIIT Pet Dataset Models identify the breed of a cat or dog from a photograph. Meta measures breed classification across 37 categories. | University of Oxford and IIIT Hyderabad | General visual questions | School | Not in cards | |||||
| PACT (robot arm orchestration) Agents write and revise programs to control a simulated robot arm. The simulator checks whether the arm completes the requested manipulation. | Aryan Madhav Verma | Simulated robots | Expert | Not in cards | Sep 2026 | ||||
| PaintSkills Models generate images of specified objects, counts, and spatial relationships. Detectors check whether an image satisfies the requested composition. | Jaemin Cho and colleagues | Image generation and editing | Everyday | Not in cards | Feb 2022 | ||||
| Paired-symmetry suite Models respond to paired political prompts. Writer compares refusal behavior across the paired requests. | Writer | Bias, fairness and politics | Everyday | Not in cards | Aug 2026 | ||||
| PalmX 2025 General Culture Models choose answers about customs, history and daily life across Arab countries. The test contains 2,000 questions in Modern Standard Arabic. | University of British Columbia and Qatar Computing Research Institute | Knowledge in many languages | School | Not in cards | Jun 2025 | ||||
| PalmX 2025 Islamic Culture Models choose answers about Islamic practices, scripture and history in Modern Standard Arabic. The organizers score exact answer choices. | University of British Columbia and Qatar Computing Research Institute | Knowledge in many languages | School | Not in cards | Jun 2025 | ||||
| Palmyra X6 Enterprise Workflow Evaluations Agents perform production workflows that require retrieval, tool use, content generation, or delegation. Writer compares capability and cost. | Writer | Running a business | Expert | Not in cards | Aug 2026 | ||||
| Paloma Language models are measured by perplexity on held-out text from 585 domains, from news sites to individual subreddits. | Allen Institute for AI | Commonsense | Everyday | Not in cards | Dec 2023 | ||||
| PAM-Audio Models judge the quality of generated environmental sounds and how closely each recording matches its caption. The listening study used 100 AudioCaps captions. | Carnegie Mellon University and Microsoft | Audio understanding and voice | Everyday | Not in cards | Feb 2024 | ||||
| Parafraseja Models decide whether two Catalan sentences express the same meaning. Human annotators wrote contrasting pairs. | Barcelona Supercomputing Center | Reading in many languages | School | Not in cards | Oct 2022 | ||||
| ParaShoot Models answer Hebrew questions by selecting words from a Hebrew Wikipedia passage. Scorers compare the selected text with human answers. | Tel Aviv University | Reading in many languages | School | Not in cards | Sep 2021 | ||||
| Pare-Bench Proactive assistants infer user goals and act across stateful digital applications with simulated users and multi-app task environments. | Apple | Apps and MCP servers | Expert | Not in cards | Apr 2026 | ||||
| Parley Labs negotiation tasks Agents negotiate contracts and plan around market changes with scripted counterparties. An offline verifier grades outcomes. Ten public development tasks accompany private scored sets. | parley-labs | Running a business | Expert | Not in cards | Nov 2026 | ||||
| ParsiNLU Models read Persian passages, compare queries, and identify sentiment. The six tasks also cover entailment, multiple-choice questions, and translation. | Daniel Khashabi and colleagues | Reading in many languages | University | Not in cards | Oct 2020 | ||||
| ParsiNLU Mizan Literary Translation Models translate literary passages between Persian and English. ParsiNLU scores each direction against 10,000 reference translations using SacreBLEU. | ParsiNLU team | Translation | University | Not in cards | Oct 2020 | ||||
| Partner Restrictions Compliance Models identify bank partner restrictions that apply to businesses. Ramp scores missed restrictions, coverage across categories, and false positives. | Ramp Labs | Finance | Expert | Not in cards | Mar 2026 | ||||
| PARTNR Agents follow natural language instructions in human-robot collaboration scenes. Tasks require planning and coordination among multiple embodied agents. | Meta FAIR and Superintelligence Labs | Embodied reasoning | University | Not in cards | Nov 2024 | ||||
| PASCAL VOC 2012 Models locate objects and label their pixels in photographs. Meta reports detection and segmentation results on the 2012 challenge. | PASCAL VOC organizers | Detection and captioning | Everyday | Not in cards | May 2012 | ||||
| PASCAL VOC 2007 Replaced in May 2012 | Not in cards | Apr 2007 | |||||||
| PASCAL-Context Models assign a semantic class to each pixel in a photograph. The benchmark includes objects and background regions. | PASCAL-Context team | Detection and captioning | Everyday | Not in cards | Jun 2014 | ||||
| PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. | Medicine | Expert | Not in cards | Jul 2025 | |||||
| PAWBench Video generators are run many times on the same start and action, and the outcomes become a distribution over physical behaviors. The test checks whether it matches reference probabilities and covers every valid behavior. | PAWBench team (with Hugging Face's Sayak Paul) | Physical prediction | Everyday | Not in cards | Aug 2026 | ||||
| PAWS-X Models decide whether two sentences with similar wording express the same meaning across languages. | Yinfei Yang and colleagues | Reading in many languages | School | Not in cards | Aug 2019 | ||||
| PCB-Bench Models answer placement and routing questions and describe circuit boards from editor screenshots. The authors report choice accuracy and separate measures for free-form answers. | Hong Kong University of Science and Technology (Guangzhou) | Electronics | University | Not in cards | |||||
| pCLUE Models follow Chinese prompts for classification, inference, generation and reading comprehension. The leaderboard combines exact match and accuracy across tasks. | SuperCLUE | Reading in many languages | School | Not in cards | Oct 2022 | ||||
| PE-A-Frame internal audio event localization Models locate sound events within recordings. Meta compares model sizes using the area under the receiver operating characteristic curve. | Meta FAIR | Audio understanding and voice | Everyday | Not in cards | Dec 2025 | ||||
| PE-AV internal video-to-audio retrieval Models match videos to music tracks. Meta compares retrieval accuracy on internal clips that last from five to thirty seconds. | Meta FAIR | Audio understanding and voice | Everyday | Not in cards | Dec 2025 | ||||
| Penn epilepsy clinic notes Models read dated paragraphs from epilepsy clinic notes and classify recent seizures. Two other questions ask for seizure frequency and the most recent seizure date. | University of Pennsylvania | Medicine | University | Not in cards | Jan 2022 | ||||
| People’s Daily and Children’s Fairy Tale Models fill a missing word using a Chinese news passage or a fairy tale. The fairy tales test whether a reader trained on news can transfer. | iFLYTEK Research and Harbin Institute of Technology | Reading in many languages | School | Not in cards | Jul 2016 | ||||
| PersistBench Models answer requests using stored memories. A judge measures inappropriate disclosure, agreement with false user beliefs and failure to use relevant memories. | Sidharth Pulipaka and colleagues | Deception and alignment | Everyday | Not in cards | Feb 2026 | ||||
| Personal Agent Bench Personal assistants complete everyday tasks using connected accounts. Experts check whether each assistant completes the work and respects the user’s instructions. | micro1 | Documents and spreadsheets | Everyday | Not in cards | |||||
| PestCLEF 2026 Models extract structured facts from plant health documents. Organizers compare the predicted relationships with expert annotations on a held-out test set. | INRAE and Paris-Saclay University | Reading comprehension | University | Not in cards | Sep 2025 | ||||
| PFBench Models answer questions about a power grid after specified changes to loads, generators, or transmission lines. The grader checks structured answers against power flow calculations. | Buxin She | Electronics | University | Not in cards | Mar 2026 | ||||
| PHEB Models answer multiple-choice questions from Portuguese national secondary school exams administered between 2006 and 2023. | PHEB authors | Knowledge in many languages | School | Not in cards | Jan 2026 | ||||
| PHEE Models extract adverse effects and potential therapeutic effects from published medical cases. They identify the event triggers and details such as the treatment and the patient. | University of Warwick and Northeastern University | Medicine | University | Not in cards | Oct 2022 | ||||
| PHINC Models translate social media posts that mix Hindi and English into English. | Vivek Srivastava and colleagues | Translation | School | Not in cards | Apr 2020 | ||||
| PhoATIS Models identify a traveler’s intent and label relevant words in Vietnamese air travel queries. HELM uses the intent classification task. | Mai Hoang Dao, Thinh Hung Truong, and Dat Quoc Nguyen | Reading in many languages | Everyday | Not in cards | Aug 2021 | ||||
| PhoMT Models translate between Vietnamese and English. Held-out sentence pairs provide reference translations in both directions. | Long Doan and colleagues | Translation | University | Not in cards | Nov 2021 | ||||
| PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. | Meta FAIR and Superintelligence Labs | Physical prediction | School | Not in cards | May 2020 | ||||
| PhysCapBench Models caption videos of physical interactions. The benchmark checks which motions, contacts, and changes of state the captions describe across 1,500 clips. | Peking University, Tsinghua University, Beihang University, and collaborators | Video captioning | Everyday | Not in cards | Nov 2026 | ||||
| PhysGym Arena DR-Hard Models repair Gym simulators that must behave correctly under unseen physical parameters. The grader compares their simulations with hidden tests and reference trajectories. | kishanpb | Scientific code | Expert | Not in cards | May 2026 | ||||
| Physics GRE (Inflection) Models answer questions from Physics GRE exams. Inflection reports majority-vote accuracy, while the harness also supports direct multiple-choice scoring. | Inflection AI | Science questions | University | Not in cards | Mar 2024 | ||||
| Physiome-ODE Models forecast irregularly sampled biological time series generated from published differential equations. The authors compare forecasts using mean squared error. | Christian Klötergens and colleagues | Genomics and life sciences | University | Not in cards | Feb 2025 | ||||
| PhyWorldBench Video generators create scenes that follow physical laws across object motion, energy conservation, rigid interactions and human or animal motion. | NVIDIA Cosmos Lab | Video generation | Expert | Not in cards | Feb 2026 | ||||
| Pico DSL v0 Models execute a small string transformation language without tools. An interpreter checks the final string across 240 generated examples. | oso | Language reasoning | School | Not in cards | Jun 2026 | ||||
| PII-Masking-300k Models locate and classify personal information in multilingual synthetic text. The open portion covers six languages. The authors separately describe a finance and insurance portion. | Ai4Privacy | Following precise instructions | Everyday | Not in cards | Mar 2024 | ||||
| PiiClustering Models group synthetic English financial documents by document type. The labels include bank statements, payment confirmations, and insurance claim forms. | The Hong Kong University of Science and Technology | Finance | Everyday | Not in cards | Jul 2024 | ||||
| PiRhDy Accompaniment Suggestion Models rank 50 accompaniment phrases for a MIDI melody. Several tracks can belong with the melody. The authors report mean average precision. | Nankai University, National University of Singapore, A*STAR, and Tsinghua University | Audio understanding and voice | School | Not in cards | Oct 2020 | ||||
| PiRhDy Melody Completion Models rank 50 possible continuations of a MIDI melody. One phrase follows the prompt in the original piece. The authors report mean average precision. | Nankai University, National University of Singapore, A*STAR, and Tsinghua University | Audio understanding and voice | School | Not in cards | Oct 2020 | ||||
| PitVQA Models answer questions about a frame from pituitary surgery. Questions cover the surgical step, the instruments in view, and what the surgeon is doing. | University College London and collaborators | Medicine | Expert | Not in cards | May 2024 | ||||
| Pixel2Perspective Models locate and segment objects from aerial and ground views. Tests vary the camera angle, weather, and use of visible or infrared images. | Indian Institute of Science and collaborators | Detection and captioning | Everyday | Not in cards | Nov 2026 | ||||
| PLABA 2024 Complete Abstract Adaptation Models rewrite 400 new biomedical abstracts for patients. Reviewers grade how well the rewrites preserve information and explain it clearly. | U.S. National Library of Medicine and National Institute of Standards and Technology | Medicine | University | Not in cards | Jan 2024 | ||||
| PLABA 2023 Abstract Adaptation Replaced in Jan 2024 | Not in cards | May 2023 | |||||||
| PLABA 2024 Term Replacement Models identify difficult medical terms, classify how to adapt them, and supply wording for patients. The organizers held out 300 abstracts. | U.S. National Library of Medicine and National Institute of Standards and Technology | Medicine | University | Not in cards | Aug 2024 | ||||
| Places Audio Captions (Hindi) Models retrieve photographs from spoken Hindi descriptions or retrieve descriptions from photographs. The score counts how often the correct match appears among the top results. | MIT Spoken Language Systems Group | Audio understanding and voice | Everyday | Not in cards | Apr 2018 | ||||
| Places205 Models classify photographs by scene type. Meta compares scene recognition on the 205-category edition of Places. | MIT Computer Science and Artificial Intelligence Laboratory | General visual questions | Everyday | Not in cards | |||||
| Plain English Summarization of Contracts Models summarize contract clauses in plain English. Human summaries provide references for evaluating how well each response conveys the legal language. | Laura Manor and colleagues | Law | University | Not in cards | Jun 2019 | ||||
| Plain Language Adaptation of Biomedical Abstracts Models rewrite biomedical abstracts in plain language for people who asked health questions. The authors collected 750 abstracts for 75 questions. | U.S. National Library of Medicine | Medicine | University | Not in cards | Jul 2022 | ||||
| PlayWorld An agent player pursues objectives inside video world models. Gemini 3.1 Pro applies weighted rubrics for geometry, interaction, and state evolution across the resulting videos. | University of Hong Kong and Kuaishou Technology | Interactive world models | Everyday | Not in cards | Aug 2026 | ||||
| PMPP CUDA Evaluation Models implement 53 CUDA kernels and answer 146 questions about parallel programming. The tasks follow the textbook Programming Massively Parallel Processors. | Sinatras | GPU kernels | University | Not in cards | Oct 2025 | ||||
| PNCExtract Models read a research paper and list its polymer nanocomposite samples. They link each sample to its matrix, filler, and composition by mass or volume. | Duke University | Lab and data work | University | Not in cards | Feb 2024 | ||||
| Poem Sentiment Models judge the sentiment expressed in individual verses of English poems from Project Gutenberg. | Emily Sheng and colleagues | Creative writing | School | Not in cards | Dec 2020 | ||||
| PoemSum Models summarize the meaning of an English poem. The authors compare each summary with a reference interpretation using ROUGE and BERTScore. | Islamic University of Technology, York University, and Dialpad Canada | Reading comprehension | School | Not in cards | Dec 2023 | ||||
| Polaris Healthcare Safety Benchmarks Models conduct simulated healthcare conversations. The evaluation measures clinical safety, communication, and administrative task performance. | Hippocratic AI | Medicine | Expert | Not in cards | |||||
| PolEmo 2.0 Models classify the sentiment of Polish consumer reviews. Separate test sets measure performance in familiar and unfamiliar domains. | Jan Kocon and colleagues | Reading in many languages | Everyday | Not in cards | Nov 2019 | ||||
| PolEval 2025 Gender-inclusive Proofreading Models rewrite Polish passages that use generic masculine forms while preserving their meaning. | Institute of Computer Science, Polish Academy of Sciences | Reading in many languages | School | Not in cards | May 2025 | ||||
| PolEval 2025 Gender-sensitive Translation Models translate inclusive Polish passages into English and English passages into gender-inclusive Polish. | Institute of Computer Science, Polish Academy of Sciences | Translation | University | Not in cards | May 2025 | ||||
| Polish Speech Emotion Recognition Challenge Models identify one of six emotions in Polish speech after training on recordings in other languages. | PolEval 2025 Task 4 organizers | Audio understanding and voice | Everyday | Not in cards | May 2025 | ||||
| PopCS Models transcribe lyrics from Chinese pop singing. Qwen compares recognition errors across speech models. | PopCS team | Speech recognition and translation | Everyday | Not in cards | May 2021 | ||||
| Post-OCR Text Correction in Sanskrit Models correct machine transcriptions of printed Sanskrit books in Devanagari. The benchmark reports character and word error rates. | Ayush Maheshwari, Nikhil Singh, Amrith Krishna, and Ganesh Ramakrishnan | Reading in many languages | University | Not in cards | Jun 2022 | ||||
| PosterBench Agents turn research papers into posters. Programmatic checks and visual model judgments score seven content and design dimensions, with ceilings for serious rendering or layout failures. | Meituan, MBZUAI, Huazhong University of Science and Technology and others | Documents and spreadsheets | University | Not in cards | Aug 2026 | ||||
| PosteriorBench Samplers generate rare model behaviors and are scored by hit rate, posterior divergence, and inference cost. | Transluce | AI research tasks | Research frontier | Not in cards | Nov 2026 | ||||
| PPR10K Models retouch groups of portraits against three expert retouchers’ Adobe Camera Raw edits. The test measures accuracy on people and color consistency across each group. | The Hong Kong Polytechnic University and Alibaba DAMO Academy | Image generation and editing | University | Not in cards | Mar 2021 | ||||
| PPTCoEditBench Agents edit selected content in PowerPoint slides. Tasks include changing a title or table while preserving surrounding shapes and animations. | Tencent | Documents and spreadsheets | School | Not in cards | Nov 2026 | ||||
| PPTFixBench Agents inspect a PowerPoint file and repair its formatting and content. Each of the 50 files contains more than 25 issues. | Tencent | Documents and spreadsheets | University | Not in cards | Nov 2026 | ||||
| PreScience Models forecast future scientific contributions through collaborator, prior-work, contribution and impact prediction tasks. | Ai2 and University of Chicago | Real-world events | Research frontier | Not in cards | Feb 2026 | ||||
| PretrainGym Agents select and mix public training data for a fixed language model. The grader trains on 400 million tokens and rewards low test loss. It penalizes overlap with evaluation data. | Justin Yang Chae | Machine learning engineering | Research frontier | Not in cards | Oct 2026 | ||||
| Privy Synthetic PII Protocol Trace Dataset Models locate personal information in synthetic API messages represented as protocol traces. The small and large configurations have separate test files. | Benjamin Kilimnik and Pixie | Reading comprehension | University | Not in cards | Sep 2022 | ||||
| ProbSum 2023 Models list active diagnoses and problems from hospital progress notes. Organizers compare the lists with a physician’s annotations using ROUGE-L F1. | University of Wisconsin and collaborators | Medicine | Expert | Not in cards | Jan 2023 | ||||
| PST50 Models transfer a reference look to a photograph. The paired test compares 50 outputs with reference grades that professionals refined in DaVinci Resolve. | Nanyang Technological University and SenseTime Research | Image generation and editing | University | Not in cards | Jun 2025 | ||||
| PsychoBench Models answer questionnaires from 13 psychological scales. Each instrument produces a profile of traits, relationships, motivations or emotional abilities. | PsychoBench authors | Deception and alignment | Everyday | Not in cards | Oct 2023 | ||||
| PsyCoMark: Conspiracy Detection Models decide whether a Reddit comment expresses a conspiracy belief. The organizers score predictions against human labels using weighted F1. | PsyCoMark organizers | Reading comprehension | School | Not in cards | Mar 2025 | ||||
| PsyCoMark: Conspiracy Marker Extraction Models mark the words that describe conspiracy claims in Reddit comments. The organizers score overlap with human annotations across five marker types. | PsyCoMark organizers | Reading comprehension | School | Not in cards | Mar 2025 | ||||
| PsyTAR Models identify adverse reactions in patient reviews of four antidepressants. The original corpus also labels withdrawal symptoms, reasons for treatment, and reported effectiveness. | Maryam Zolnoori and colleagues | Medicine | University | Not in cards | Jan 2019 | ||||
| PUBHEALTH Models classify public health claims as true, false, mixed or unproven. The original benchmark also asks models to explain their verdicts. | Neema Kotonya and Francesca Toni | Faithful to sources | University | Not in cards | Nov 2020 | ||||
| Public Tests, Hidden Tests Models write Python functions from public test cases. The grader runs hidden tests to measure whether each solution generalizes beyond the visible examples. | summer-winter-98379 | Function-level problems | School | Not in cards | May 2026 | ||||
| PubMed causal language use Models classify the wording of a research conclusion as a direct causal claim, a conditional causal claim, a correlation, or no relationship. | Syracuse University and Jun Wang | Medicine | University | Not in cards | Sep 2019 | ||||
| PubMed-PICO Models classify sentences in structured medical abstracts. Section headings supply the reference labels, including those for the study population, intervention, and outcome. | MIT | Medicine | University | Not in cards | Jun 2018 | ||||
| PUCRS inpatient fall detection Models identify whether a Portuguese progress note documents a fall. Nursing students labeled 355 of the 1,078 notes as fall related. | Pontifical Catholic University of Rio Grande do Sul | Medicine | University | Not in cards | Jul 2019 | ||||
| PUMaC 2024 Models solve problems from the 2024 Princeton University Mathematics Competition. Scores measure the percentage of contest problems answered correctly. | Princeton University Mathematics Competition | Competition problems with answers | Competition | Not in cards | Nov 2024 | ||||
| PUMaVOS Models track masks for object parts across video frames. A prompt can identify a small region, such as one cheek of a person's face. | PUMaVOS team | Video understanding | Everyday | Not in cards | Jul 2023 | ||||
| PushWorld Agents move shapes through puzzles that require tools or pushing obstacles aside. The benchmark includes more than 200 puzzles with goal positions. | Google DeepMind | Puzzles and chess | School | Not in cards | Jan 2023 | ||||
| PVD Bench Models match captions with short videos in both retrieval directions. Meta measures retrieval on held-out video and caption pairs. | Meta FAIR | Video understanding | Everyday | Not in cards | Apr 2025 | ||||
| PWS-Decisions Decision APIs choose answers to 300 closed-ended questions whose answers depend on current or scattered facts on the public web. | Parallel Web Systems | Deep research | Expert | Not in cards | Nov 2026 | ||||
| PWS-Legal legal Web research systems answer dated questions about companies, laws, regulations, intellectual property, court decisions, and legal documents. | Parallel Web Systems | Law | Expert | Not in cards | Nov 2026 | ||||
| PWS-Health health Replaced in Nov 2026 | Medicine | Not in cards | Nov 2026 | ||||||
| PWS-Finance finance Replaced in Nov 2026 | Finance | Not in cards | Oct 2026 | ||||||
| Py Bug Trace Level 3 Models fix Python functions after inspecting examples that miss the defect. The grader runs a stricter hidden test suite. | neomatrix369 | Function-level problems | University | Not in cards | Jun 2026 | ||||
| Q-Bench Image-Pair Perception Models answer multiple-choice questions comparing low-level attributes across image pairs. The score is overall answer accuracy in the A1-Pair table. | Nanyang Technological University and collaborators | General visual questions | Everyday | Not in cards | Sep 2023 | ||||
| Q-Bench Single-Image Perception Models answer multiple-choice questions about low-level attributes of individual images. The score is overall answer accuracy in the A1-Single table. | Nanyang Technological University and collaborators | General visual questions | Everyday | Not in cards | Sep 2023 | ||||
| Q2D-Chunk Retrievers find short passages that answer a query or supply the evidence needed to interpret the answer. | Perplexity | Questions that need search | University | Not in cards | Oct 2026 | ||||
| QA4MRE Models read documents and choose answers that require integrating information across the text. The harness includes the 2011, 2012 and 2013 CLEF tests. | CLEF QA4MRE organizers | Reading comprehension | University | Not in cards | |||||
| QAG-360K Models compare two satellite images, answer questions about land cover, and mark the relevant pixels. The authors score answer accuracy and mask overlap separately. | Ke Li and colleagues | Spatial reasoning and counting | School | Not in cards | Oct 2024 | ||||
| QALB 2015 Models correct Arabic news comments and essays by Arabic learners. The organizers score edits separately for native writers and learners. | Qatar Arabic Language Bank project | Reading in many languages | School | Not in cards | Feb 2015 | ||||
| QALB 2014 Replaced in Feb 2015 | Not in cards | Oct 2014 | |||||||
| QAMPARI Models return multiple answers to a question using evidence scattered across Wikipedia passages. Each question has at least five correct answers. | Tel Aviv University | Questions that need search | University | Not in cards | May 2022 | ||||
| QASC Models answer eight-choice grade-school science questions by combining supporting facts from a large text corpus. Accuracy checks the selected answer against the answer key. | Allen Institute for Artificial Intelligence and collaborators | Science questions | School | Not in cards | Oct 2019 | ||||
| QBQTC Models judge how relevant a page title is to a Chinese search query. | CLUE benchmark team | Questions that need search | Everyday | Not in cards | |||||
| QEVD-FIT-COACH Models watch a workout and decide when to give feedback. The benchmark scores feedback timing and the content of the coaching advice. | Qualcomm AI Research | Video understanding | Everyday | Not in cards | Jul 2024 | ||||
| QIAS 2025 Islamic Knowledge Assessment Models answer Arabic multiple-choice questions about classical Islamic texts. The released test contains 1,000 rows and 729 distinct questions. | QIAS organizers | Exam questions | University | Not in cards | May 2025 | ||||
| QMSum Models answer a query about a long meeting transcript with a written summary of the relevant parts. The 1,808 query and summary pairs cover 232 meetings from several domains. Scores compare outputs with reference summaries using ROUGE. | Yale University (LILY Lab) and collaborators | Reasoning over long documents | School | Not in cards | Apr 2021 | ||||
| Qodo AI Code Review Benchmark Models review real merged pull requests with planted functional bugs and best-practice violations. Findings are matched to the planted issues and scored by F1. | Qodo | Fixing real repositories | Expert | Not in cards | Feb 2026 | ||||
| Quartets Audio-to-Score Transcription Models transcribe synthesized string quartets by Haydn, Mozart, and Beethoven into four-part scores. The released test split contains 7,611 excerpts. | University of Alicante | Audio understanding and voice | Expert | Not in cards | Oct 2019 | ||||
| QuartetsMIREX 2026 Audio-to-Score Transcription Models transcribe synthesized string quartet recordings into notation. Organizers scored 373 excerpts by Schubert, Brahms, and Mendelssohn, whose music the original Quartets dataset omits. | MIREX | Audio understanding and voice | Expert | Not in cards | |||||
| Questflow Traders Arena Agents and people trade on Hyperliquid and Polymarket in a public competition. The leaderboard reports weekly portfolio returns alongside gains and trading volume. | Questflow | Finance | University | Not in cards | Jun 2026 | ||||
| QuixBugs Models repair 40 classic algorithm programs, in Python and Java, that each contain a one-line bug. The bugs fall into 14 defect classes. A fix counts when the program passes the test cases. | MIT (Lin, Koppel, Chen, Solar-Lezama) | Function-level problems | School | Not in cards | Jul 2017 | ||||
| Qur’an QA 2023 Passage Retrieval Models rank Quran passages that answer an Arabic question or return no passages. Organizers score mean average precision at ten. | Qatar University | Questions that need search | University | Not in cards | Jun 2023 | ||||
| Qur’an QA 2023 Reading Comprehension Models extract answers from a supplied Quran passage or return no answer. Organizers give partial credit for overlapping answer spans. | Qatar University | Reading comprehension | School | Not in cards | Jun 2023 | ||||
| Qwen-Audio-Agent In-House Cockpit Benchmark Agents handle requests with cockpit tools. The evaluation measures whether they complete each task when they call tools directly, delegate, or choose between both routes. | Alibaba Token Foundry | Function calling | Everyday | Not in cards | Sep 2026 | ||||
| Qwen-Music Cover Song Generation Models remake a reference melody using requested lyrics and style. The evaluation aligns the melodies and measures their average pitch difference in semitones. | Alibaba Qwen Team | Audio understanding and voice | University | Not in cards | Jul 2026 | ||||
| Qwen3 forced alignment benchmarks Models place word or character boundaries in audio. Qwen measures timestamp shifts against automatic and human reference alignments. | Qwen | Speech recognition and translation | School | Not in cards | Jan 2026 | ||||
| Qwen3-ASR internal transcription benchmarks Models transcribe accented conversations, noisy speech, tongue twisters, and multilingual news. Qwen reports error rates for each recording condition. | Qwen | Speech recognition and translation | Everyday | Not in cards | Jan 2026 | ||||
| RaceBias (Race-Based Medicine) Models answer clinical questions used to identify unsupported race-based medical beliefs. Evaluators check responses for those beliefs. | Jesutofunmi A. Omiye and colleagues | Bias, fairness and politics | Expert | Not in cards | Oct 2023 | ||||
| RadNLP 2024 Lung Cancer Staging Models assign detailed lung cancer stages to radiology reports. The organizers score Japanese and English separately and require all three stage labels to match. | NTCIR RadNLP organizers | Medicine | Expert | Not in cards | Mar 2024 | ||||
| NTCIR-17 Radiology Report TNM Staging Replaced in Mar 2024 | NTCIR MedNLP-SC organizers | Not in cards | Sep 2022 | ||||||
| RadNLP 2024 Radiology Report Segmentation Models label sentences in lung cancer reports with eight categories, including tumor size and distant metastasis. The organizers score labels using micro F2. | NTCIR RadNLP organizers | Medicine | Expert | Not in cards | Jun 2024 | ||||
| RAGBench Evaluators judge whether retrieved passages are relevant and generated answers are grounded. Galileo compares their judgments with reference annotations. | Galileo | Faithful to sources | University | Not in cards | Jun 2024 | ||||
| Railhead Agents run a transport company in OpenTTD. The evaluator replays the action log and lets the company run without intervention before measuring its performance. | Rachit Shah | Video games | Everyday | Not in cards | Oct 2026 | ||||
| Range Repair Agent Agents repair a Python function that merges touching ranges. The sole task earns one point if the edited function passes every held-out test. | akashdeepjassal | Function-level problems | School | Not in cards | Oct 2026 | ||||
| Rauh Bundestag sentence sentiment validation Models rate the sentiment of individual German parliamentary sentences. Three native speakers independently rated all 1,500 sentences from Bundestag speeches delivered from 1991 through 2013. | Christian Rauh | Reading in many languages | Everyday | Not in cards | May 2018 | ||||
| RAVDESS Models identify an actor's expressed emotion from recorded speech or song. | Steven Livingstone and Frank Russo | Speech recognition and translation | Everyday | Not in cards | May 2018 | ||||
| Ravnursson Faroese Speech and Transcripts Models transcribe 3,002 Faroese recordings from 20 test speakers. The speakers read aloud for four and a half hours. | Reykjavík University and the Ravnur Project | Speech recognition and translation | Everyday | Not in cards | Sep 2022 | ||||
| RCTs to Scale nudge forecasts Models predict how each of 14 government interventions changes participation or compliance relative to a control. Researchers correlate predictions with observed changes in percentage points. | Stefano DellaVigna and Elizabeth Linos | Social simulation | University | Not in cards | Jul 2020 | ||||
| Re:Baltica Latvian and Russian chatbot sources Chatbots answer four questions about the Baltic states in Latvian and Russian. Journalists count the propaganda sources each chatbot offers in each language. | Re:Baltica | Long-form factuality | Everyday | Not in cards | Sep 2025 | ||||
| ReactSim-Bench Driving simulators predict how traffic responds when the vehicle changes its behavior. The authors test those reactions in 2,636 scenarios. | Zhiyuan Zhang and colleagues | Simulated robots | University | Not in cards | Jun 2026 | ||||
| RealCommBench Models distinguish human and synthetic speech after transmission over real phone calls. Paired recordings measure how the call changes detection error. | Li Wang and colleagues | Speech recognition and translation | Everyday | Not in cards | Oct 2026 | ||||
| RealEstate10K Models estimate relative camera poses across frames from real estate videos. The benchmark supplies camera trajectories for comparison. | Tinghui Zhou and colleagues | Spatial reasoning and counting | School | Not in cards | |||||
| RealHall Evaluators score model responses for hallucinations. Four datasets cover questions with and without reference documents. The score measures agreement with binary ground truth labels. | Galileo | Hallucination rates | University | Not in cards | Oct 2023 | ||||
| RealHarm Safety systems classify real-world harmful conversations and compare their decisions with documented safe and unsafe versions across ten harm categories. | Giskard AI | Refusing harmful requests | Expert | Not in cards | Apr 2025 | ||||
| RealityTest Conversational models answer identity questions in text and speech across five languages. A judge classifies explicit, ambiguous and missing identity disclosure. | UK AI Security Institute | Deception and alignment | Everyday | Not in cards | Jun 2026 | ||||
| Reasoning Gym Models solve procedurally generated reasoning problems with automatic rewards. Harbor draws three easy and three hard instances from each of 96 task generators. | Reasoning Gym team | Language reasoning | School | Not in cards | Jan 2025 | ||||
| ReasoningJudgeBench Judge models rate reasoning outputs to measure evaluator reliability and agreement across difficult tasks. | Salesforce AI Research | AI research tasks | Expert | Not in cards | Jan 2026 | ||||
| ReasonSeg Models infer which object an instruction describes and return its mask. Instructions can require reasoning about the image. | ReasonSeg team | Detection and captioning | Everyday | Not in cards | Aug 2023 | ||||
| ReasonVOS Models segment and track objects in videos from explicit referring queries and implicit reasoning queries. The score is overall J&F on ReVOS. | University of Technology Sydney and Alibaba (Yan et al., VISA) | Video understanding | Everyday | Not in cards | Jul 2024 | ||||
| Recipe1M image and recipe retrieval Models match food photographs with recipes. Google evaluates image retrieval of the matching ingredients and instructions separately. | Recipe1M team | General visual questions | Everyday | Not in cards | |||||
| Recovery-Bench Agents repair a task after a weaker agent leaves failed actions and a corrupted environment. The score is the share of recovery tasks that pass the original task verifier. | Letta | Long terminal sessions | Expert | Not in cards | Aug 2025 | ||||
| Red Team Attack Simulator Models test whether a simulated assistant follows its safety policy across three conversational turns. The evaluation records policy violations and penalizes direct harm. | Intertwine | Jailbreaks and prompt injection | University | Not in cards | Sep 2025 | ||||
| Red-Teaming Resistance Benchmark Models respond to prompts from multiple safety datasets. Judges assess whether the responses comply with harmful requests. | Haize Labs and Hugging Face | Jailbreaks and prompt injection | Everyday | Not in cards | Feb 2024 | ||||
| RedlineBench v2 (Fa1zvn) Agents negotiate eight contract terms with a simulated vendor. The grader compares the agreement with the best deal both sides could accept. | fa1zvn | Running a business | University | Not in cards | Jun 2026 | ||||
| Ref-DAVIS17 Models segment and track the object that a sentence describes through each frame of a DAVIS 2017 video. Scored by J&F, the mean of mask overlap and contour accuracy. | Max Planck Institute for Informatics (Khoreva, Rohrbach, Schiele) | Video understanding | Everyday | Not in cards | Mar 2018 | ||||
| Ref-YouTube-VOS Models segment and track the object that a sentence describes in YouTube-VOS videos. Scored by J&F on a server-graded validation split. | Seoul National University and Adobe Research (Seo, Lee, Han, URVOS) | Video understanding | Everyday | Not in cards | Aug 2020 | ||||
| Ref-ZOM Models mark the pixels described by a phrase in a COCO image. A phrase can refer to zero, one or several objects. | Yutao Hu and colleagues | Detection and captioning | Everyday | Not in cards | Oct 2023 | ||||
| RefAV Models find driving scenarios described in natural language. They identify when each scenario occurs and track the objects involved. | Carnegie Mellon University | Detection and captioning | University | Not in cards | Mar 2025 | ||||
| RefChecker Models generate answers that are checked for fine-grained claims against references. Claim-level labels measure factual consistency and hallucination detection quality. | Amazon Science | Hallucination rates | University | Not in cards | Jan 2024 | ||||
| RefineBench Models refine answers to 1,002 problems across 11 domains using checklists that specify required content and errors. | NVIDIA Research Taiwan | Following precise instructions | Expert | Not in cards | Nov 2025 | ||||
| RefSpatial-Expand-Bench Location Models solve spatial location questions in the RefSpatial expansion set. The score is the published overall accuracy. | BAAI (RoboRefer) | Embodied reasoning | Everyday | Not in cards | Oct 2025 | ||||
| RefSpatial-Expand-Bench Placement Models solve spatial placement questions in the RefSpatial expansion set. The score is the published overall accuracy. | BAAI (RoboRefer) | Embodied reasoning | Everyday | Not in cards | Oct 2025 | ||||
| RegexEval Models turn natural-language descriptions into regular expressions. The dataset contains 762 patterns with matching and nonmatching examples. The hub port holds out 62 patterns. | Mohammed Latif Siddiq and colleagues | Function-level problems | University | Not in cards | Nov 2023 | ||||
| RELATE Models listen to an audio clip and predict how well it matches a caption. Human listeners also judge which sounds occur and in what order. | Yusuke Kanamori and colleagues | Audio understanding and voice | Everyday | Not in cards | Jun 2025 | ||||
| ReMuQ Models retrieve a knowledge passage that answers a question about an image. | Man Luo and colleagues | General visual questions | School | Not in cards | Jun 2023 | ||||
| RepliBench Agents acquire resources, exfiltrate model weights, replicate onto compute and persist over time. Task checks measure components of autonomous self-replication. | UK AI Security Institute | Labs' internal R&D tests | Research frontier | Not in cards | Apr 2025 | ||||
| Research Plan Generation Models write research plans from goals derived from scientific papers. Judges grade each plan against criteria specific to its source paper. | Meta | AI research tasks | Expert | Not in cards | Dec 2025 | ||||
| RESPIN-S1.0 Models transcribe dialect speakers reading sentences about farming and finance in nine Indian languages. The test holds out speakers and reports errors by dialect. | Indian Institute of Science and Navana Tech | Speech recognition and translation | Everyday | Not in cards | May 2025 | ||||
| REVERSAL-BENCH Reset-free reinforcement learning agents learn manipulation policies across environments with controlled reversibility and recoverability checks. | Apple | Simulated robots | Expert | Not in cards | Sep 2026 | ||||
| Reverse Wikipedia Paragraphs Models reverse a Wikipedia paragraph character by character. The evaluator compares the response with the exact reversal using the longest common subsequence. | d42me | Following precise instructions | School | Not in cards | Oct 2025 | ||||
| Revisited Oxford and Paris Models retrieve photographs of the same landmark from Oxford and Paris image collections. The revised benchmark defines medium and hard evaluation protocols. | Filip Radenović and colleagues | General visual questions | Everyday | Not in cards | Mar 2018 | ||||
| Reward Hack Bench Agents attempt eight coding and puzzle tasks under controlled conditions. A judge distinguishes independent solutions from use of published answers. The control omits hints. | Islo Labs | Deception and alignment | University | Not in cards | Jun 2026 | ||||
| Reward-seeking via contrastive belief updates Researchers compare model decisions under different beliefs about what a grader rewards. The evaluation measures how often those beliefs change behavior. | Apollo Research and OpenAI | Deception and alignment | Expert | Not in cards | Jul 2026 | ||||
| RICH Models reconstruct people interacting with their surroundings. The evaluation compares body joints and mesh vertices with the reference geometry. | RICH team | Spatial reasoning and counting | School | Not in cards | Jun 2022 | ||||
| RivaBench Models answer questions about stand-up routines and academic talks using speech and visual cues. They also classify clips as real or synthetic. | University of Cambridge, ByteDance, and Tsinghua University | Video understanding | University | Not in cards | Feb 2025 | ||||
| RLBench Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates. | Imperial College London (Dyson Robotics Lab) | Simulated robots | Everyday | Not in cards | Sep 2019 | ||||
| RMU:ECHR Models mark argument spans in European Court of Human Rights rulings and identify the type of argument and the party making it. | Ivan Habernal and colleagues | Law | University | Not in cards | Aug 2022 | ||||
| Roark Live Bench v1.3 Voice agents answer calls over the public phone network and use simulated business tools. Roark scores task completion, tool use, speech quality, and response time. | Roark | Customer service | University | Not in cards | Oct 2026 | ||||
| Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. | Robocurve | Real robots | Everyday | Not in cards | |||||
| Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. | Robocurve | Real robots | Everyday | Not in cards | Oct 2026 | ||||
| Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. | Sharpa + BitRobot/FrodoBots + Nippon Origami Association + partners | Real robots | Everyday | Not in cards | Jun 2026 | ||||
| Robots Evolved GUI lessons Agents use screenshots, clicks, and keys to move sliding robots onto a goal. Ten puzzles enforce a move budget and offer no undo. | suryanshsh | Puzzles and chess | Everyday | Not in cards | Oct 2026 | ||||
| Robust Multi-view Depth Models estimate depth from multiple photographs of a scene. The benchmark compares relative depth errors across datasets from different environments. | Robust Multi-view Depth team | Spatial reasoning and counting | School | Not in cards | Sep 2022 | ||||
| robust-kbench Agents write CUDA kernels that are tested across several weight initializations and input shapes, so shape and magnitude exploits fail. Correctness checks and profiling score each kernel. | Sakana AI | GPU kernels | Expert | Not in cards | Sep 2025 | ||||
| ROI Calculator (dsm) Models calculate total and annualized investment returns, decide whether to invest, and explain the decision. The evaluator checks each answer against a bundled reference. | dsm | Finance | School | Not in cards | Feb 2026 | ||||
| RollerCoasterTycoonBench Models manage a theme park by choosing weekly actions in RollerCoaster Tycoon 2. | Vals AI | Video games | Everyday | Not in cards | Nov 2026 | ||||
| Roman Urdu Code-Switch Benchmark Models answer 35 items mixing Roman Urdu and English. Tasks cover comprehension, sentiment, translation, and Pakistani cultural context. | UMAR SHAREEF | Reading in many languages | School | Not in cards | Oct 2026 | ||||
| Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. | Google Research | Simulated robots | Everyday | Not in cards | Oct 2020 | ||||
| ROOMELSA Models use a marked region in a room panorama and a written description to retrieve the matching 3D object. They return their ten closest matches. | Vietnam National University Ho Chi Minh City and University of Dayton | Spatial reasoning and counting | Everyday | Not in cards | Mar 2025 | ||||
| RouterBench Routers select a language model for each prompt using cached responses, quality measurements, and costs. Evaluation measures the quality and cost of those selections. | Martian | Composite indices | University | Not in cards | |||||
| RP2K Models identify retail products from photographs taken in stores. The test distinguishes products with similar packaging. | RP2K team | General visual questions | Everyday | Not in cards | Jun 2020 | ||||
| RRG24 Models write findings and impressions from one or more chest X-rays per study. Organizers score each section separately using five metrics. | Stanford AIMI and collaborators | Medicine | Expert | Not in cards | Feb 2024 | ||||
| RRN Models summarize Icelandic news articles from the national broadcaster RÚV using reference summaries. | Þór Sverrisson and Hafsteinn Einarsson | Reading in many languages | School | Not in cards | May 2023 | ||||
| RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. | OpenAI | Labs' internal R&D tests | Research frontier | Not in cards | Jul 2026 | ||||
| RSICD Models describe aerial scenes in a sentence. The authors provide five reference captions per image and reserve 1,093 images for testing. | Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li | Detection and captioning | Everyday | Not in cards | Sep 2017 | ||||
| RSIEval Models describe scenes in aerial photographs and answer questions about objects, positions, and road directions. Human graders score captions and answers separately. | Yuan Hu and colleagues | General visual questions | Everyday | Not in cards | Jul 2023 | ||||
| RTP-LX Models continue or judge toxic prompts hand-translated from RealToxicityPrompts into dozens of languages, with native-speaker labels. Measures toxic output and toxicity detection. | Microsoft (de Wynter et al.) | Bias, fairness and politics | Everyday | Not in cards | Apr 2024 | ||||
| RTSGameBench Agents play the open real-time strategy game Beyond All Reason. | RTSGameBench authors | Video games | Expert | Not in cards | Jun 2026 | ||||
| Rundkast Models transcribe Norwegian radio news, interviews and debates. Word error rate measures transcription accuracy. | Norwegian University of Science and Technology | Speech recognition and translation | Everyday | Not in cards | May 2008 | ||||
| Runway Model Router Benchmark Video generation systems produce clips for 250 image-to-video prompts across ten production categories and are compared by usable rate and cost. | Runway | Video generation | Expert | Not in cards | Sep 2026 | ||||
| RVMS-Bench Agents search open-domain videos and localize moments using global impression, key moment, temporal context and auditory memory cues. | Tencent | Web browsing | Expert | Not in cards | Mar 2026 | ||||
| S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. | ByteDance Seed | AI research tasks | Expert | Not in cards | Aug 2026 | ||||
| SA-1B Hard human pose evaluation Models locate body and foot joints in photographs with unusual viewpoints, poses, and occlusions. Meta measures the fraction of correctly placed joints. | Meta | Spatial reasoning and counting | School | Not in cards | Feb 2026 | ||||
| SA-37 interactive segmentation Models refine an object mask after a user clicks on an image. The benchmark averages overlap across datasets at different numbers of clicks. | Meta | Detection and captioning | Everyday | Not in cards | Jul 2024 | ||||
| Saarland MIDI-Audio Piano Music Models transcribe student piano performances. The Disklavier captured the reference notes as the musicians played. | Saarland University, Max Planck Institute for Informatics, and Hochschule für Musik Saar | Audio understanding and voice | Expert | Not in cards | Oct 2011 | ||||
| Sakana Japanese-English translation evaluation Models translate Japanese text into English while handling Japan-specific proper nouns and honorifics. | Sakana AI | Translation | University | Not in cards | Aug 2026 | ||||
| Sakana kana-letter reading-order set Model recovers the reading order of 25 expert-annotated classical Japanese kana letters (chirashigaki). Scored by normalized edit distance. | Sakana AI | Documents, charts and text in images | Expert | Not in cards | Jun 2026 | ||||
| SalesBench (Insurance Sales) Agents find insurance leads, quote plans, and schedule callbacks in a simulated sales pipeline. The evaluator rewards converted monthly premiums and checks compliance with calling restrictions. | SalesBench | Running a business | University | Not in cards | Feb 2026 | ||||
| SalesBench-100 Agents reconcile sales evidence, update authorized records, and deliver executive briefs across 100 workflows. Checks grade procedure, system state, and outputs. | Blobfish AI | Apps and MCP servers | Expert | Not in cards | Aug 2026 | ||||
| SAM 3D Artist Objects Models reconstruct an object's shape and placement from a single photograph. The benchmark compares predictions with meshes created by 3D artists. | Meta | Spatial reasoning and counting | School | Not in cards | Nov 2025 | ||||
| SAM 3D Body synthetic pose evaluation Models estimate human pose and shape from synthetic images of individual people. Meta compares predicted joints and mesh vertices with the rendering parameters. | Meta | Spatial reasoning and counting | School | Not in cards | Feb 2026 | ||||
| SAM Audio Judge evaluation Models rate the quality of separated speech, music, and environmental sounds. Meta measures correlation with listener ratings for completeness, precision, faithfulness, and overall quality. | Meta | Audio understanding and voice | Everyday | Not in cards | Dec 2025 | ||||
| Samrómur 21.05 Models transcribe Icelandic sentences read aloud by volunteers. The training, development, and test sets use different speakers. | Reykjavík University | Speech recognition and translation | Everyday | Not in cards | May 2021 | ||||
| Samrómur Children Models transcribe Icelandic speech from children reading prompts. The test covers girls and boys aged six through sixteen. | Reykjavík University | Speech recognition and translation | Everyday | Not in cards | Sep 2021 | ||||
| SandboxEscapeBench Agents attempt to escape a container sandbox and retrieve a flag from the host. Inspect runs the challenge inside reproducible virtual machines. | UK AI Security Institute | Finding and exploiting real bugs | Expert | Not in cards | Oct 2026 | ||||
| SandhiKosh Models join Sanskrit words under sound change rules or recover the original words from a joined form. The authors score accuracy separately for joining and splitting. | Indian Institute of Technology Delhi and IBM Research | Reading in many languages | University | Not in cards | May 2018 | ||||
| SatLidar canopy height evaluation Models predict canopy height from satellite images. Meta compares height estimates with airborne lidar measurements and reports errors on held-out regions. | Meta | Spatial reasoning and counting | University | Not in cards | Aug 2025 | ||||
| SatNet Agents assign ground antennas and contact times to spacecraft requests across five weeks of NASA Deep Space Network operations. They balance unmet demand across missions. | NASA Jet Propulsion Laboratory and collaborators | Tasks from real jobs | University | Not in cards | Nov 2021 | ||||
| SAUDADE-PT Models determine the chronological order of events related to Portuguese history and culture. | AMALIA project | Reasoning in many languages | School | Not in cards | Apr 2026 | ||||
| Sayfullina Soft Skill Span Extraction Models mark the words that express a soft skill in English job ads. The test scores exact span matches. | IT University of Copenhagen and LMU Munich | Other professions | Everyday | Not in cards | Sep 2022 | ||||
| SB10k Models classify the sentiment of German social media posts using labels from three human annotators. | Mark Cieliebak and colleagues | Reading in many languages | Everyday | Not in cards | Apr 2017 | ||||
| SC-CoMIcs Models read superconductivity abstracts and identify materials, properties, and measurements. The corpus also records relationships, including which dopant replaces an element. | Toyota Technological Institute and Toyota Central R&D Labs | Lab and data work | University | Not in cards | May 2020 | ||||
| ScanNet Models infer indoor scene geometry from images. Meta evaluates depth predictions against measurements from scanned indoor environments. | Angela Dai and colleagues | Spatial reasoning and counting | School | Not in cards | Feb 2017 | ||||
| ScanNet-1500 Models match features between pairs of indoor views and estimate relative camera poses. The evaluation measures pose accuracy on 1,500 image pairs. | Paul-Edouard Sarlin and colleagues | Spatial reasoning and counting | School | Not in cards | Nov 2019 | ||||
| ScanNet++ v2 Models reconstruct indoor scenes from images of real scans. The second release adds scene types and updates the annotations and camera poses. | ScanNet++ team | Spatial reasoning and counting | School | Not in cards | Dec 2024 | ||||
| ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change. | IBM Research | Fixing real repositories | Expert | Not in cards | May 2026 | ||||
| SceneFun3D Models identify movable parts in indoor scans, locate the part needed for a written task, and estimate how it moves. | ETH Zürich and collaborators | Spatial reasoning and counting | Everyday | Not in cards | Mar 2024 | ||||
| Sci Pipeline v1 Agents clean hourly sensor readings and write daily means, record counts, and extremes to a JSON file. The grader checks the saved file. | afzal0 | Data science and SQL | University | Not in cards | Sep 2026 | ||||
| SciCap Models describe figures from scientific publications. Researchers compare generated captions with the text that accompanies each figure. | Ting-Yao Hsu and colleagues | Documents, charts and text in images | School | Not in cards | Oct 2021 | ||||
| SciDocs Models compare representations of scientific papers on citation prediction, recommendation, and classification tasks. | Allen Institute for AI | Research science | University | Not in cards | Apr 2020 | ||||
| Science Sim Chemistry Models compare simulated compounds and choose whether to advance or reject them. The evaluator scores decisions against measured reaction outcomes and a budget. | allan | Lab and data work | University | Not in cards | May 2026 | ||||
| SciFact Models retrieve research abstracts and determine whether they support or refute a scientific claim. | Allen Institute for AI | Faithful to sources | Expert | Not in cards | Apr 2020 | ||||
| SciQ Models answer crowdsourced multiple-choice science exam questions in physics, chemistry and biology, each with a supporting passage. | Allen Institute for AI (Welbl et al.) | Science questions | School | Not in cards | Jul 2017 | ||||
| SciVid Video models answer questions about scientific videos across several disciplines. The benchmark tests whether models recognize procedures, objects, and events in specialized footage. | Google DeepMind | Vision | University | Not in cards | Jul 2025 | ||||
| SCORE (Consistency and Robustness) Models answer the same questions under different option orders, prompt formats and decoding settings. The evaluator measures consistency as well as accuracy. | NVIDIA | Following precise instructions | University | Not in cards | Nov 2024 | ||||
| Screen2Words Models summarize the purpose and contents of a mobile interface from a screenshot. The benchmark uses human-written summaries as references. | Google Research | Documents, charts and text in images | School | Not in cards | Oct 2021 | ||||
| SCROLLS Models do seven long-text tasks such as summarization, question answering and natural language inference over naturally long documents. | Tel Aviv University and collaborators | Reasoning over long documents | University | Not in cards | Jan 2022 | ||||
| SD-QA Models answer spoken questions recorded in different English dialects. The benchmark compares answer accuracy across regional varieties of English. | Fahim Faisal and colleagues | Audio understanding and voice | Everyday | Not in cards | Sep 2021 | ||||
| SDBench Diagnostic agents order tests, ask questions and commit diagnoses in sequential clinical cases. Scores combine diagnostic correctness with test cost. | Microsoft | Medicine | Expert | Not in cards | Oct 2026 | ||||
| SEA-HELM Audio Models answer the same reasoning and instruction-following tasks from speech and text in five languages. Identical checks measure the accuracy lost with speech input. | AI Singapore | Speech recognition and translation | School | Not in cards | Oct 2026 | ||||
| Security Configuration Verification Agents audit Kubernetes and Terraform configurations, identify security violations and submit fixes. The grader weights detected violations by severity and validates patches. | Intertwine | Secure code and review | University | Not in cards | Sep 2025 | ||||
| Sedgewick and Wayne Algorithms Exercises Models solve 469 exercises from the first six chapters of Algorithms, fourth edition. A Java sandbox tests code, and a model judge scores theoretical answers. | Amit Prakash | Function-level problems | University | Not in cards | Nov 2025 | ||||
| Segment Anything Video Models predict masks for prompted objects throughout a video. The evaluation compares predicted boundaries and regions with annotated object tracks. | Meta | Video understanding | Everyday | Not in cards | Jul 2024 | ||||
| Segment Anything with Concepts Models find every instance of a concept named in a prompt. The benchmark compares image masks and video tracks with human annotations. | Meta | Detection and captioning | Everyday | Not in cards | Nov 2025 | ||||
| Self-Instruct User-Oriented Instructions Models follow manually written instructions on everyday tasks. Evaluators judge whether their responses meet the user's request. | Yizhong Wang and colleagues | Following precise instructions | Everyday | Not in cards | Dec 2022 | ||||
| SemClinBr Models identify and classify medical terms in Brazilian Portuguese clinical notes. The corpus contains 1,000 notes from multiple specialties. | Pontifical Catholic University of Paraná and collaborators | Medicine | University | Not in cards | Jan 2020 | ||||
| SemEval 2015 Task 14 Analysis of Clinical Text Models identify disorder mentions and map them to terminology codes in clinical notes. A separate task asks for attributes such as negation and severity. | SemEval Task 14 and ShARe organizers | Medicine | University | Not in cards | May 2014 | ||||
| SemEval 2014 Task 7 Analysis of Clinical Text Replaced in May 2014 | SemEval Task 7 and ShARe organizers | Not in cards | Nov 2013 | ||||||
| ShARe/CLEF eHealth 2013 Task 1 Replaced in Nov 2013 | ShARe/CLEF eHealth organizers | Not in cards | Feb 2013 | ||||||
| SemEval 2023 Task 3: Genre, Framing and Persuasion Techniques in Online News Models classify a news article as reporting, opinion, or satire. They also label its frames and identify persuasion techniques in individual paragraphs. | SemEval 2023 Task 3 organizers | Reading in many languages | University | Not in cards | Sep 2022 | ||||
| SemEval 2026 Task 4: Narrative Story Similarity and Narrative Representation Learning Models choose which story is more similar to a reference story. A second track compares story embeddings. Each track scores accuracy on 400 triples. | University of Hamburg, Toloka AI, German UDS and TU Darmstadt | Reading comprehension | School | Not in cards | Jul 2025 | ||||
| SemEval-2015 Task 1 Models decide whether two short social media posts express the same meaning. | Wei Xu and colleagues | Reading comprehension | School | Not in cards | Jun 2015 | ||||
| SemiKongBench-100 Agents resolve 100 synthetic semiconductor operating exceptions. The evaluator grades each workflow with deterministic checks. | Blobfish AI | Tasks from real jobs | Expert | Not in cards | Sep 2026 | ||||
| Sentences Involving Compositional Knowledge Models compare sentences whose meanings change with their grammatical structure. The dataset includes both relatedness ratings and entailment labels. | Marco Marelli and colleagues | Reading comprehension | School | Not in cards | May 2014 | ||||
| SentinelBench Agents monitor changing web environments and respond when scripted events make progress possible. Scores measure completion, reaction time and resource use. | Microsoft Research | Web browsing | Expert | Not in cards | Jun 2026 | ||||
| Sequential Question Answering Models answer sequences of related questions about Wikipedia tables. Answers identify table cells, so later questions can depend on earlier selections. Evaluation checks predicted answers against the annotated cells. | Microsoft Research | Reading comprehension | School | Not in cards | Nov 2016 | ||||
| SEvenLLM Models analyze cyber security incidents and write responses. Tasks split into understanding and generation over 28 subcategories, in Chinese and English. | SEvenLLM authors (Ji et al.) | Finding and exploiting real bugs | Expert | Not in cards | May 2024 | ||||
| SF20K Competition 2025 Models answer story questions about short films. GPT-4.1-nano compares their answers with human references. Films released during the competition determine the final rankings. | SLoMO Workshop at ICCV 2025 | Long videos | Everyday | Not in cards | Jul 2025 | ||||
| SFI-Bench Multimodal models answer video questions about spatial layouts, object affordances and functional reasoning in indoor scenes. | Apple | Spatial reasoning and counting | University | Not in cards | May 2026 | ||||
| SHREC 2026 Human Perceived Visual Complexity of 3D Shapes Models rank CAD parts, primate teeth, and fractals by visual complexity. People compared pairs of CAD parts and fractals. A zoological measure ranks the teeth. | SHREC 2026 visual complexity track organizers | Spatial reasoning and counting | Everyday | Not in cards | |||||
| SHROOM-Visions 2026 Models inspect an image and a written response. They mark unsupported characters, estimate the probability of an error and classify each error in four languages. | SHROOM-Visions organizers | Faithful to sources | Everyday | Not in cards | Apr 2026 | ||||
| SICK-NL Models judge whether one Dutch sentence entails, contradicts, or is neutral toward another sentence. | Gijs Wijnholds and Michael Moortgat | Reasoning in many languages | School | Not in cards | Apr 2021 | ||||
| SIGHT (Search and Intent Gauge for Hospitality and Tourism) Agents recommend restaurants, hotels and attractions from user preferences and search results. Evaluators check constraints, evidence and privacy. | AGI-Eval (Meituan) | Apps and MCP servers | Everyday | Not in cards | May 2026 | ||||
| SIGHUM Sanskrit Word Segmentation Models recover individual Sanskrit words from continuous text. Evaluators measure word precision, recall, F1, and the percentage of sentences whose entire segmentation matches. | Amrith Krishna, Pavankumar Satuluri, and Pawan Goyal | Reading in many languages | University | Not in cards | Jun 2017 | ||||
| SIGMORPHON 2023 Shared Task on Interlinear Glossing Models label each word’s meaningful parts with translations or grammatical tags. The organizers rank submissions by word-level accuracy pooled across languages. | SIGMORPHON | Translation | Expert | Not in cards | Feb 2023 | ||||
| SignBD-Word Models identify isolated Bangla signs from video or body pose. The authors score top one and top five accuracy. | Bangladesh University of Engineering and Technology | Video understanding | Everyday | Not in cards | Jun 2022 | ||||
| Silent Exit Models fix four typos without changing correct lines. A held-out test measures whether models learned to quit early to avoid penalties. | dp-learning-rl | Deception and alignment | Everyday | Not in cards | Jul 2026 | ||||
| Simple Cooccurrence Bias Models complete occupation prompts with gendered words. The harness reports the percentage of prompts where the model prefers a male term. | OpenAI and EleutherAI | Bias, fairness and politics | Everyday | Not in cards | May 2020 | ||||
| Simreal ML Benchmark ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them. | Simreal | Machine learning engineering | Expert | Not in cards | Sep 2026 | ||||
| SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. | HCLTech | CAD models | Expert | Not in cards | Oct 2026 | ||||
| Singapore Factuality Models answer questions about Singapore's laws, economy, and public policies. A model judge checks whether each answer contradicts the reference. | AI Verify Foundation | Short factual questions | University | Not in cards | Aug 2026 | ||||
| Singapore POFMA Statements Models classify statements as true or false using claims from Singapore's official corrections. The test includes true statements to check models that always answer false. | AI Verify Foundation | Short factual questions | Everyday | Not in cards | Jan 2025 | ||||
| Singapore Safety Questions Models answer 59 open-ended questions about culturally or legally sensitive situations in Singapore. A Llama Guard classifier checks whether each response is safe. | AI Verify Foundation | Refusing harmful requests | Everyday | Not in cards | May 2024 | ||||
| Singapore Undesirable Content Models answer prompts about sensitive situations in Singapore. A classifier labels each response and reports the percentage it judges safe. | AI Verify Foundation | Refusing harmful requests | Everyday | Not in cards | Aug 2026 | ||||
| Single-turn election integrity (Anthropic, 2026) Models answer legitimate election requests and decline requests that violate Anthropic’s election policy. Graders check refusals and harmful completions separately. | Anthropic | Refusing harmful requests | Everyday | Not in cards | Apr 2026 | ||||
| Election Questions (Anthropic) v2024 Replaced in Apr 2026 | Not in cards | Jun 2024 | |||||||
| SingMOS Models listen to Chinese and Japanese singing and predict how listeners rate its quality. The recordings include human voices and synthesized voices. | Renmin University of China and Carnegie Mellon University | Audio understanding and voice | Everyday | Not in cards | Jun 2024 | ||||
| SkillSpan Models mark the words that describe skills or knowledge in English job postings. Exact span matches determine precision, recall, and F1. | IT University of Copenhagen and LMU Munich | Other professions | School | Not in cards | Apr 2022 | ||||
| SkillSpan-ESCO Models link skills mentioned in job advertisements to entries in the European skills classification. Human annotators assigned the reference labels. | TechWolf and Ghent University | Other professions | University | Not in cards | Sep 2022 | ||||
| skLEP Models classify Slovak reviews, recognize entailment, identify names, and answer questions about Slovak Wikipedia passages. | Marek Suppa and colleagues | Reading in many languages | School | Not in cards | Jul 2025 | ||||
| SLDBench Agents infer scaling laws from data covering more than 5,000 language model experiments. Eight tasks test how well their formulas predict results at other scales. | Haowei Lin and colleagues | AI research tasks | Expert | Not in cards | Jul 2025 | ||||
| SlideVQA Models answer questions about whole slide decks, with single-hop, multi-hop and numerical reasoning. The set has 14.5k questions over 2.6k decks. | NTT (Tanaka, Nishida et al.) | Documents, charts and text in images | Everyday | Not in cards | Jan 2023 | ||||
| SLIP zero-shot transfer suite Models assign labels to images using text descriptions of the classes. The evaluation averages zero-shot accuracy across 26 recognition tasks. | Norman Mu and colleagues | General visual questions | Everyday | Not in cards | Dec 2021 | ||||
| Slitherlink (savi) Models place edges to form a single loop on seven-by-seven Slitherlink boards. The evaluation uses ten boards disjoint from its 15 training examples. | savi | Puzzles and chess | School | Not in cards | May 2026 | ||||
| SLoMO Audio Description Competition 2026 Models write one sentence for each gap in a film’s dialogue. Judges score how well the descriptions convey visual information and match professional references. | SLoMO Workshop at ECCV 2026 | Video captioning | University | Not in cards | Jun 2026 | ||||
| SLPHelm UltraSuitePlus Models analyze recordings of children’s speech for speech sound disorders. The suite also includes transcription and symptom classification tasks. | SAA Lab and SLPHelm contributors | Speech recognition and translation | Expert | Not in cards | Sep 2025 | ||||
| SLURP Models identify the intent and named entities in spoken requests. NVIDIA evaluates both tasks within AIR-Bench. | Emanuele Bastianelli and colleagues | Audio understanding and voice | Everyday | Not in cards | Nov 2020 | ||||
| SMEBench Models edit recorded music according to instructions. The benchmark uses paired source and target recordings to check changes to pitch, instrumentation, articulation, and dynamics. | Matthieu Cervera and colleagues | Audio understanding and voice | University | Not in cards | Sep 2026 | ||||
| ŚMIGIEL Models classify Polish texts as human or machine written. The test includes genres and generators absent from training. | PolEval 2025 organizers | Reading in many languages | University | Not in cards | Apr 2025 | ||||
| SMILE Video Laugh Reasoning Models explain why people laugh during sitcoms and TED talks. The authors compare generated explanations with human references. | POSTECH and collaborators | Video understanding | Everyday | Not in cards | Dec 2023 | ||||
| SMM4H 2025 Shingles Vaccine Reactions Models classify English Reddit posts about shingles vaccines. Organizers score F1 for posts in which the author reports a personal reaction. | SMM4H-HeaRD organizers | Medicine | Everyday | Not in cards | Feb 2025 | ||||
| SMM4H-HeaRD 2026 Flu Vaccine Effectiveness Models classify posts by the author’s flu vaccination status and flu test result. The labels distinguish the current flu season from past seasons. | SMM4H-HeaRD organizers | Medicine | Everyday | Not in cards | Nov 2025 | ||||
| SMM4H-HeaRD 2026 MedSynth Dialogue-to-Note Models turn synthetic clinical dialogues into notes with four sections. The organizers generated 368 separate test dialogues and scored the notes against references. | Dalhousie University and collaborators | Medicine | University | Not in cards | Nov 2025 | ||||
| SMM4H-HeaRD 2026 Multilingual Adverse Drug Event Detection Models flag personal reports of drug side effects in seven languages. The organizers withheld Farsi from training and added 15,184 Farsi posts at test time. | SMM4H-HeaRD organizers | Medicine | Everyday | Not in cards | Nov 2025 | ||||
| SMM4H-HeaRD 2025 Multilingual Adverse Drug Event Detection Replaced in Nov 2025 | Not in cards | Feb 2025 | |||||||
| SMS Spam Collection Models classify 5,574 English text messages as spam or legitimate. The Prime port creates an 80/20 split and adds risk and reasoning scores. | Tiago Almeida, José María Gómez Hidalgo and Akebo Yamakami | Overall chat quality | Everyday | Not in cards | Sep 2011 | ||||
| Snowflake AI_EXTRACT Document Extraction Evaluation Models extract fields, lists and tables across 2,789 annotated tasks from 11 datasets. Missing items, extra items and incorrect cells reduce the score. | Snowflake | Documents, charts and text in images | University | Not in cards | Nov 2026 | ||||
| Snowflake Analytical Search: ArXiv Papers Agents count, group and compare findings across research papers. The evaluation checks numeric and categorical answers against references and scores lists with F1. | Snowflake | Deep research | University | Not in cards | Jun 2026 | ||||
| Snowflake Arctic Enterprise Intelligence Models write code, generate SQL and follow instructions. Snowflake averages the coding, SQL and instruction-following scores. | Snowflake | Composite indices | University | Not in cards | Apr 2024 | ||||
| Snowflake Financial Filings RAG Evaluation Systems answer about 500 questions over 23,000 SEC filings. Annotators identify supporting passages, pages and answers. | Snowflake | Finance | Expert | Not in cards | Mar 2025 | ||||
| Snowflake Financial PDF Table Parsing Models reconstruct tables from 50 manually annotated financial documents. Evaluators compare complete tables, table dimensions and individual cells with reference Markdown. | Snowflake | Documents, charts and text in images | University | Not in cards | Feb 2026 | ||||
| Snowflake Multimodal PDF Retrieval Evaluation Retrievers find relevant pages in technical manuals, SEC filings and presentation slides. The tasks include 143 chart questions and 69 text questions about manuals. | Snowflake | Documents, charts and text in images | University | Not in cards | Apr 2025 | ||||
| SO-Bench Multimodal models produce schema-compliant structured outputs grounded in images from four visual domains. Static checks and rubric-based grading score semantic correctness and schema compliance. | Apple | Documents, charts and text in images | University | Not in cards | Dec 2025 | ||||
| SoccerNet-Caption Models write timestamped commentary for soccer broadcasts. The set has almost 37k commentaries over 715.9 hours of video. Streaming reports also score when the model chooses to speak. | University of Liege and KAUST (SoccerNet) | Video captioning | Everyday | Not in cards | Apr 2023 | ||||
| Social Science Experiment Treatment Effect Archive Models predict how survey interventions change responses in the United States. Researchers compare predicted treatment effects with experimental estimates using Pearson correlation. | Ashwini Ashokkumar, Luke Hewitt, Isaias Ghezae, and Robb Willer | Social simulation | University | Not in cards | Aug 2024 | ||||
| SocialMaze Models perform social deduction and other interactive tasks under deep reasoning and uncertainty. | SocialMaze authors | Multi-agent games | Everyday | Not in cards | Nov 2026 | ||||
| SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. | King's College London, Fudan University and The Alan Turing Institute | Deception and alignment | Everyday | Not in cards | Jun 2026 | ||||
| SoCRATES Models mediate disputes in 600 scenarios across eight domains. A model judge rates topic agreement to measure consensus gain, intervention timeliness, and intervention effectiveness. | KAIST and Chungnam National University | Multi-turn conversation | University | Not in cards | Jun 2026 | ||||
| SOFC-Exp Models identify sentences that report fuel cell experiments, mark materials and measurements, and assign roles such as anode material or operating temperature. | Bosch Center for Artificial Intelligence and Robert Bosch Corporate Research | Lab and data work | University | Not in cards | Jun 2020 | ||||
| SokoBench Models solve Sokoban puzzles that require long-horizon planning and persistent state tracking. | SokoBench authors | Puzzles and chess | Everyday | Not in cards | Jan 2026 | ||||
| SOL-ExecBench Systems write CUDA kernels for 235 workloads taken from real model architectures, run on B200 GPUs. The SOL score measures how much of the gap between a baseline and the analytic hardware speed limit a kernel closes. | NVIDIA | GPU kernels | Expert | Not in cards | Mar 2026 | ||||
| Solubility Optimization (anindya) Models choose chemical transformations to improve predicted solubility. The package includes a benchmark runner that measures improvement on evaluation examples. | anindya | Lab and data work | University | Not in cards | Dec 2025 | ||||
| SongBench Models generate songs from lyrics and a style prompt. A trained music judge scores seven aspects, including the melody, arrangement, vocals, and mix. | Tencent and Tsinghua University | Audio understanding and voice | University | Not in cards | Mar 2026 | ||||
| SongEval Models predict professional listeners' ratings for full-length songs in English and Mandarin. The test covers memorability, phrasing, structure, coherence, and musicality. | Northwestern Polytechnical University and collaborators | Audio understanding and voice | University | Not in cards | May 2025 | ||||
| Soohak Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems. | OneLineAI, EleutherAI and CMU with 64 mathematicians (Guijin Son et al.) | Research mathematics | Research frontier | Not in cards | May 2026 | ||||
| SOP-Bench Agents follow expert-written standard operating procedures across 12 industrial domains, using executable tool interfaces over more than 2,000 tasks with 10 to 50 decision points. | Amazon | Running a business | Expert | Not in cards | Feb 2026 | ||||
| SoundingEarth audio and image retrieval (MTEB) Models match a field recording to an aerial image of the same location, or retrieve the recording from the image. Each direction has 1,000 queries and 2,048 candidates. | Massive Text Embedding Benchmark contributors | Audio understanding and voice | Everyday | Not in cards | Jul 2026 | ||||
| SPACE Models solve spatial cognition tasks involving maps, object layouts, spatial attention and memory in text and image presentations. Accuracy measures task performance. | Apple | Spatial reasoning and counting | University | Not in cards | Apr 2025 | ||||
| SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). | xAI (SpaceXAI) | Labs' internal R&D tests | Expert | Not in cards | |||||
| SPair-71k Models match corresponding parts of objects across image pairs. The benchmark includes changes in viewpoint, scale, and occlusion. | Juhong Min and colleagues | Spatial reasoning and counting | School | Not in cards | Aug 2019 | ||||
| SpartQA Models answer questions about spatial relationships described in a passage. | Roshanak Mirzaee and colleagues | Language reasoning | School | Not in cards | Apr 2021 | ||||
| Spatial-Gym Models find paths that satisfy grid puzzle rules. The evaluation uses 500 SPaRC puzzles in one-shot and stepwise settings, including a setting that allows backtracking. | Lukas Kaesberg and collaborators | Puzzles and chess | School | Not in cards | Apr 2026 | ||||
| SpatialBench (Ropedia) Spatial foundation models are tested across many 3D tasks, viewpoints, scene domains, input densities and hardware limits, beyond the domains they were trained for. | Ropedia | Spatial reasoning and counting | Everyday | Not in cards | May 2026 | ||||
| SpatialGenEval Text-to-image models generate scenes from 1,230 prompts, and vision-language judges answer 12,300 spatial questions about the resulting images. | AMAP and Beijing University of Posts and Telecommunications | Image generation and editing | University | Not in cards | Jan 2026 | ||||
| Spec Gaming Review Agents review 12 programs that pass their test suites and decide whether each meets its specification. Six programs exploit gaps in the tests. | fredrik-talent | Function-level problems | University | Not in cards | Sep 2026 | ||||
| Spec2Control Models convert plant control requirements into function block diagrams. The evaluation compares control strategies, connections, and alarm mappings with reviewed reference diagrams. | ABB | Electronics | Expert | Not in cards | Oct 2025 | ||||
| SpecDecode RL Agents choose how many tokens to draft during speculative decoding. An offline grader measures speedup over the target model while requiring identical output. | semih | Machine learning engineering | Expert | Not in cards | Jul 2026 | ||||
| Speech Robust Bench Models transcribe speech under different recording distortions. Word error rates show how each distortion affects recognition. | Muhammad A. Shah and colleagues | Speech recognition and translation | Everyday | Not in cards | Mar 2024 | ||||
| SpeechIO ASR leaderboard Models transcribe Mandarin recordings from the SpeechIO evaluation sets. Qwen reports an aggregate transcription error. | SpeechIO | Speech recognition and translation | Everyday | Not in cards | |||||
| SpeechMMLU Speech-language models answer MMLU-derived questions from text or synthesized speech in text-to-text, speech-to-text, text-to-speech and speech-to-speech settings. | Xiaomi MiMo | Audio understanding and voice | University | Not in cards | Sep 2025 | ||||
| SpeedrunBench Agents try to finish goals in 10 video games as fast as possible. Score is time to goal in frames or seconds, lower is better. | Patronus AI | Video games | Everyday | Not in cards | Sep 2026 | ||||
| SpireBench Agents play Slay the Spire and are scored on long-horizon deck-building decisions. | SpireBench authors | Video games | Everyday | Not in cards | Oct 2026 | ||||
| SPOT5 daily photograph scheduling Agents choose which photographs a satellite should take and assign cameras. Feasible schedules earn points for image priority while respecting instrument, data rate, and storage limits. | Centre national d'études spatiales and collaborators | Tasks from real jobs | University | Not in cards | |||||
| SPY Models locate personal information about the author of a synthetic medical consultation or legal question. Other people mentioned in the text fall outside the annotation policy. | Moscow Institute of Physics and Technology, MTS AI, ITMO University and AIRI | Following precise instructions | Everyday | Not in cards | Sep 2024 | ||||
| SQAD Models find the answer to a Czech question in a Wikipedia passage. Student annotators identify the answer sentence and answer span. | Natural Language Processing Centre at Masaryk University | Reading in many languages | School | Not in cards | |||||
| SQL-API-Bench Agents answer questions that require both database queries and API calls. The evaluator checks answers across two mixed database and API benchmark settings. | IBM Research | Function calling | Expert | Not in cards | Oct 2025 | ||||
| Sregym env Agents diagnose incidents in a running service with generated logs and traffic. The grader checks service recovery, the underlying fault and collateral damage. | ericxu88 | Apps and MCP servers | University | Not in cards | Aug 2026 | ||||
| Sri Lankan local election chatbot audit Chatbots answer 16 procedural and two political questions in English, Sinhala, and Tamil. Reviewers score procedural accuracy and political neutrality separately. | Democracy Reporting International and Factum | Long-form factuality | Everyday | Not in cards | May 2025 | ||||
| ST-VQA Models answer questions by reading text in a scene. The benchmark compares each answer with the text required by the question. | ST-VQA team | Documents, charts and text in images | School | Not in cards | May 2019 | ||||
| Stanford Cars Models identify a car's make, model, and year from a photograph. The benchmark measures fine-grained vehicle recognition. | Stanford University | General visual questions | University | Not in cards | Dec 2013 | ||||
| Stanford Online Products Models retrieve photographs of the same product. The test compares learned image features on products outside the training set. | Hyun Oh Song and colleagues | General visual questions | Everyday | Not in cards | Nov 2015 | ||||
| STAR situated video reasoning Models answer multiple-choice questions about how people interact with objects in videos. The questions cover interactions, sequences, predictions, and feasibility. | Bo Wu and colleagues | Video understanding | Everyday | Not in cards | |||||
| STaRK Retrieval systems find the right items for complex queries over knowledge bases that mix free text with relations between entities. The three domains are product search, academic paper search and precision medicine. Most queries are synthesized and checked by people, and a smaller set is written by people. | Stanford University (SNAP group) and collaborators | Questions that need search | University | Not in cards | Apr 2024 | ||||
| StateBench Models follow variables through long runs of assignments and report a final value. Four tiers share one source trace and differ only in the update: fixed values, reassignment, self-referential updates, and assignment by reference. Problems come in several formats, such as code, spreadsheet edits and people moving between rooms, at lengths from under 2K to over 2M tokens. | Ai2 | Reasoning over long documents | Everyday | Not in cards | Nov 2026 | ||||
| States United voter questions ChatGPT and Google answer voter questions. Researchers check factual claims against official sources and assess the completeness of written answers. | States United Democracy Center | Long-form factuality | Everyday | Not in cards | Aug 2026 | ||||
| SteerBench Speech models respond to spoken instructions about tone, persona, style, and speed. A judge checks their speech and words against binary rubrics. | Scale AI and the University of Maryland | Audio understanding and voice | Everyday | Not in cards | Sep 2026 | ||||
| Step-Caption Audio models describe speakers' vocal characteristics across language, age, rate, rhythm, pitch, timbre, emotion and related attributes. A judge compares the description with expert labels. | StepFun | Audio understanding and voice | University | Not in cards | Apr 2026 | ||||
| Step-DU Audio models answer targeted questions about a speaker's vocal characteristics in dialogue. A judge scores answers against reference labels. | StepFun | Audio understanding and voice | University | Not in cards | Apr 2026 | ||||
| Step-SPQA Audio models answer paralinguistic questions about age, emotion, events, gender, pitch, rhythm, scene, speed, voice sounds and style. | StepFun | Audio understanding and voice | University | Not in cards | Apr 2026 | ||||
| StepEval-Audio-360 Speech models answer multi-turn Chinese prompts about instruction following, voice understanding, reasoning, role play, creativity and speech control. | StepFun | Audio understanding and voice | University | Not in cards | Feb 2025 | ||||
| StepUp gym attendance forecasts Models predict how each four-week exercise program changes weekly gym visits relative to a placebo. Researchers compare predicted effects with recorded gym check-ins. | Katherine Milkman and the Behavior Change for Good Initiative | Social simulation | University | Not in cards | Dec 2021 | ||||
| StereoSet Models choose between stereotypical, anti-stereotypical and unrelated completions in four domains: gender, profession, race and religion. Scores combine language modeling and stereotype preference. | MIT, Intel AI and McGill University | Bias, fairness and politics | Everyday | Not in cards | Apr 2020 | ||||
| StereoTales Models write multilingual stories about characters with controlled demographic attributes. Statistical tests find overrepresented associations, and people judge their harmfulness. | Giskard AI | Bias, fairness and politics | Everyday | Not in cards | May 2026 | ||||
| STL-10 Models classify images into ten object categories using limited labeled training data. | Adam Coates and colleagues | General visual questions | Everyday | Not in cards | Jun 2011 | ||||
| STORAL Models choose which moral fits a Chinese or English story. Two generation tasks ask models to write a moral from a story or illustrate a moral. | Tsinghua University | Reading comprehension | School | Not in cards | Apr 2022 | ||||
| StreamEval Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing. | StreamEval authors | Video generation | Everyday | Not in cards | Oct 2026 | ||||
| StreamingQA Models answer time-stamped questions about news. The test measures how well they acquire new information while retaining older knowledge. | Google DeepMind | Questions that need search | Everyday | Not in cards | May 2022 | ||||
| Structured Web Data Extraction Models extract named attribute values from web pages. The benchmark compares their outputs with annotated values across website categories. | Qiang Hao and colleagues | Faithful to sources | University | Not in cards | Jul 2011 | ||||
| StyleInstructCaps Models describe a speaker's voice according to different instructions. The evaluator grades instruction following, accuracy about the recording, and consistency across the speaker's clips. | Johns Hopkins University and The University of Texas at Dallas | Speech recognition and translation | Everyday | Not in cards | Oct 2026 | ||||
| Sugar’s Kitchen AI Cooking Battle 17 Models design a summer lunch for a fictional Japanese inn. ChatGPT awards up to 40 points for the written proposal, and the organizer awards ten. | Sugar (シュガ) | Taste and style | Everyday | Not in cards | Sep 2026 | ||||
| Sumi Japanese PII Corpus Models locate personal information in Japanese business templates and prose. The release includes 2,000 template test documents and 520 documents outside those templates. | NagaYu | Reading comprehension | Everyday | Not in cards | Sep 2026 | ||||
| Summ-it Coreference Models group Portuguese noun phrases that refer to the same entity in science articles from Folha de São Paulo. | Sandra Collovini and colleagues | Reading in many languages | University | Not in cards | Jul 2007 | ||||
| SummScreenFD Models write a recap of a TV episode from its full transcript. Plot details sit in dialogue scattered across the episode, so the model must find and join them. Scores compare outputs with human-written recaps using ROUGE. | Toyota Technological Institute at Chicago and collaborators | Reasoning over long documents | School | Not in cards | Apr 2021 | ||||
| SUMO Web Claims Summarization Models write a short headline summarizing a fact-checking article. HELM uses articles about climate claims. | Rahul Mishra and colleagues | Faithful to sources | School | Not in cards | Nov 2020 | ||||
| SUN397 Models recognize the scene category in indoor and outdoor photographs. The classification task covers 397 categories. | Jianxiong Xiao and colleagues | General visual questions | Everyday | Not in cards | Jun 2010 | ||||
| Sunbird African Language Technology Models translate parallel sentences between English and six languages spoken in Uganda and East Africa. | Sunbird AI | Translation | School | Not in cards | |||||
| SUPERB Self-supervised speech models are tested as frozen encoders on a range of speech tasks, including speech recognition, keyword spotting, speaker identification, intent classification and emotion recognition. Each task trains only a light prediction head on top of the shared model. | National Taiwan University and collaborators | Speech recognition and translation | Everyday | Not in cards | May 2021 | ||||
| SuperCLUE-AISearch Search systems answer questions in Chinese. | SuperCLUE | Questions that need search | Everyday | Not in cards | |||||
| SuperCLUE-Coder Coding assistants complete programming tasks in Chinese. | SuperCLUE | Building whole projects | University | Not in cards | |||||
| SuperCLUE-CPIFOpen Models answer Chinese prompts with precise instructions. The published evaluation code checks their responses. | SuperCLUE team | Following precise instructions | Everyday | Not in cards | Jul 2025 | ||||
| SuperCLUE-DeepResearch Agents research questions and assemble reports in Chinese. | SuperCLUE | Deep research | University | Not in cards | |||||
| SuperCLUE-LarkClaw Claw assistants complete tasks through Feishu. | SuperCLUE | Apps and MCP servers | Everyday | Not in cards | |||||
| SuperCLUE-MedAssist Medical assistants respond to Chinese healthcare questions. | SuperCLUE | Medicine | Expert | Not in cards | |||||
| SuperCLUE-Office Agents process documents, analyze data, and write office materials. | SuperCLUE | Documents and spreadsheets | University | Not in cards | |||||
| SuperCLUE-R1 Search DeepSeek R1 services answer questions that require search and reasoning. | SuperCLUE | Questions that need search | University | Not in cards | |||||
| SuperCLUE-Voice (audiovisual interaction) Models respond to audio and video inputs in real time. | SuperCLUE | Audio understanding and voice | Everyday | Not in cards | |||||
| SuperCLUE-WeClaw Claw assistants complete tasks through WeChat. | SuperCLUE | Apps and MCP servers | Everyday | Not in cards | |||||
| SuperCLUE-XClaw Claw assistant products complete user tasks in Chinese. | SuperCLUE | General assistants | Everyday | Not in cards | |||||
| SuperLim 2.0 Models complete 15 Swedish language tasks across 14 datasets. Tasks include recognizing arguments, choosing synonyms, and summarizing newspaper articles. | Språkbanken Text, AI Sweden, and the National Library of Sweden | Reading in many languages | School | Not in cards | Sep 2022 | ||||
| SuperLim 1.0 Replaced in Sep 2022 | Språkbanken Text, the National Library of Sweden, RISE, and AI Sweden | Not in cards | May 2021 | ||||||
| SuperSim Models give separate scores for how similar two Swedish words are and how closely they are associated. Five Swedish speakers rated each pair. | Simon Hengchen and Nina Tahmasebi, University of Gothenburg | Reading in many languages | Everyday | Not in cards | Apr 2021 | ||||
| Surge AI blind coding evaluation Surge AI raters blindly score model coding outputs on a 1-5 scale. Mistral reports five models. | Surge AI (commissioned by Mistral) | Building whole projects | University | Not in cards | Oct 2026 | ||||
| SUST Bangla Emotional Speech Corpus Models identify which of seven emotions a speaker acts out in a Bangla sentence. | Shahjalal University of Science and Technology | Audio understanding and voice | Everyday | Not in cards | Feb 2021 | ||||
| SV-Gap Reset Release Agents write SystemVerilog control circuits from eight specifications. The evaluator tests functional behavior and checks whether each circuit obeys its reset wiring requirement. | Shraddha Sridhar | Electronics | University | Not in cards | Jul 2026 | ||||
| SVAMP Models solve 1,000 grade-school arithmetic word problems. The problems are variations of problems from earlier datasets, made to catch solvers that ignore the question or the word order. The final numeric answer is scored by exact match. | Microsoft Research India (Patel, Bhattamishra, Goyal) | Word problems | School | Not in cards | Mar 2021 | ||||
| SVG-Bench (StarVector) Models generate SVG code from images or text. The RLRF port uses the SVG-Stack image-to-SVG task and rewards the quality of the rendering. | StarVector authors | Image generation and editing | University | Not in cards | Dec 2023 | ||||
| SWAG Models read a partial description of an everyday activity. They choose the most plausible continuation from four options. | Rowan Zellers and colleagues | Commonsense | Everyday | Not in cards | Aug 2018 | ||||
| SWE-bench Models resolve 2,294 real GitHub issues from 12 Python repositories. A patch counts as resolved when the hidden repository tests pass. | Princeton University (SWE-bench team) | Fixing real repositories | Expert | Not in cards | Oct 2023 | ||||
| SWE-bench Lite Models resolve 300 self-contained GitHub issues chosen from SWE-bench to cut evaluation cost. Hidden repository tests decide each patch. | Princeton University (SWE-bench team) | Fixing real repositories | Expert | Not in cards | Mar 2024 | ||||
| SWE-Bench Pro Verified Agents repair software repositories. OpenCompass corrects inconsistent tasks and strengthens tests in SWE-Bench Pro. | OpenCompass and collaborators | Fixing real repositories | Expert | Not in cards | Sep 2026 | ||||
| SWE-gen JS Agents fix issues drawn from 1,000 merged pull requests across 30 JavaScript and TypeScript repositories. Each task includes tests that fail before the reference fix. | Abundant AI | Fixing real repositories | Expert | Not in cards | Jan 2026 | ||||
| SweDiagnostics Models judge entailment between Swedish sentences that test phenomena such as negation, quantifiers, and ambiguity. | Felix Morger, University of Gothenburg | Reasoning in many languages | School | Not in cards | Jun 2021 | ||||
| Swedish Analogy Test Set Models complete a Swedish word analogy by predicting the fourth word from the first three. Relations include grammar and geography. | Tosin Adewumi, Foteini Liwicki, and Marcus Liwicki | Reading in many languages | School | Not in cards | Jul 2020 | ||||
| SweFAQ 2.0 Models match questions from Swedish public agencies to answers from the same subject category. | Språkbanken Text, University of Gothenburg | Reading in many languages | Everyday | Not in cards | Sep 2022 | ||||
| SweNLI Models decide whether Swedish premises entail a statement, contradict it, or leave it undecided. | Språkbanken Text, University of Gothenburg | Reasoning in many languages | School | Not in cards | Feb 2023 | ||||
| SweParaphrase 2.0 Models rate meaning overlap between Swedish sentences on a scale from zero to five. | Dana Dannélls, University of Gothenburg | Reading in many languages | Everyday | Not in cards | Oct 2022 | ||||
| SweParaphrase 1.0 Replaced in Oct 2022 | Not in cards | Jun 2021 | |||||||
| SweSAT Synonyms 1.1 Models choose one of five Swedish synonyms or definitions. The official score adjusts accuracy for random guessing. | Språkbanken Text, University of Gothenburg | Reading in many languages | School | Not in cards | Mar 2023 | ||||
| SweSAT Synonyms 1.0 Replaced in Mar 2023 | Not in cards | Jun 2021 | |||||||
| SweWiC Models decide whether a highlighted Swedish word has the same meaning in two sentences. | Gerlof Bouma, University of Gothenburg | Reading in many languages | Everyday | Not in cards | Jun 2021 | ||||
| SweWinogender 2.0 Models judge whether a Swedish pronoun refers to a person in a stated occupation. The parity score checks agreement across three pronoun variants. | Språkbanken Text, University of Gothenburg | Bias, fairness and politics | Everyday | Not in cards | Feb 2023 | ||||
| SweWinograd 2.0 Models decide whether a highlighted Swedish pronoun refers to a proposed person or thing in the passage. | Språkbanken Text, University of Gothenburg | Reading in many languages | Everyday | Not in cards | Feb 2023 | ||||
| Swiss Job Market Monitor ICT Term Recognition Models mark words for information and communication technology in German-language Swiss job ads. The original test holds out 10% of the annotated ads. | University of Zurich | Other professions | University | Not in cards | May 2022 | ||||
| Swiss-Judgment-Prediction Models predict whether the Swiss Federal Supreme Court accepts or dismisses a case. | Joel Niklaus and colleagues | Law | Expert | Not in cards | Oct 2021 | ||||
| SWM-Bench (Social World Models) Predict how Kalshi and Polymarket belief trajectories shift after news events. 12k+ points across politics, finance and crypto, chronologically split. | UIUC (ulab-uiuc) | Social simulation | Expert | Not in cards | Jun 2026 | ||||
| Sydney-Captions Models describe land use in aerial images of Sydney. The dataset pairs 613 images with five human captions each. | Bo Qu, Xuelong Li, Dacheng Tao, and Xiaoqiang Lu | Detection and captioning | Everyday | Not in cards | Jul 2016 | ||||
| SympTEMIST Models locate symptoms, signs, and findings in Spanish clinical cases. They also map given mentions to SNOMED CT codes in Spanish and five additional languages. | Barcelona Supercomputing Center | Medicine | University | Not in cards | Jul 2023 | ||||
| SYNERGY+ Models rank research records against a systematic review's eligibility criteria. Reviewers' final inclusion decisions supply the reference labels. | Utrecht University and collaborators | Medicine | University | Not in cards | Aug 2026 | ||||
| SynLogic Models solve synthetic reasoning problems, including Sudoku, arithmetic puzzles, and mazes. The port uses official task graders on separate easy and hard validation sets. | MiniMax | Language reasoning | School | Not in cards | May 2025 | ||||
| Synthetic Contextual ASR Benchmark (Indic) Models transcribe a synthetic caller’s response with or without the preceding conversation. A language model scores whether each transcript preserves the caller’s intent and named entities. | Sarvam AI | Speech recognition and translation | Everyday | Not in cards | Feb 2026 | ||||
| Szeged NER Models identify named entities in Hungarian business news from the Hungarian News Agency. | György Szarvas and colleagues | Reading in many languages | School | Not in cards | May 2006 | ||||
| T2Ranking Models rank Chinese passages by their relevance to a query. | Xiaohui Xie and colleagues | Reading in many languages | School | Not in cards | Jul 2023 | ||||
| TabFact Models decide whether a statement is entailed or refuted by a Wikipedia table. Classification accuracy measures agreement with human labels. | UC Santa Barbara and Tencent AI Lab | Faithful to sources | Everyday | Not in cards | Sep 2019 | ||||
| TAC 2017 Adverse Drug Reaction Extraction from Drug Labels Models extract adverse reactions and their modifiers from prescription drug labels. They identify positive reactions and map them to standard medical terms. The organizers held out 99 labels. | Text Analysis Conference and the US National Library of Medicine | Medicine | University | Not in cards | Apr 2017 | ||||
| TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. | Shanghai Jiao Tong University and collaborators | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| TailOR Models generate images and videos of regular, unconventional, and impossible tool interactions. Annotators score instruction adherence, interaction accuracy, physical realism, and perceptual quality. | University of Illinois Urbana-Champaign, Stanford University, and Columbia University | Physical prediction | Everyday | Not in cards | Jun 2026 | ||||
| Taiwan Mandarin Speech with Video: Speech Enhancement Models use lip video to remove noise from Mandarin sentences. Researchers compare the enhanced signal with clean speech for quality and intelligibility. | Shang-Yi Chuang and colleagues, Academia Sinica and KU Leuven | Speech recognition and translation | University | Not in cards | May 2020 | ||||
| Taiwan National Dental Technician Licensing Examination 2023 Models answer 194 text-only dental technology questions from Taiwan's 2023 licensing exam. The authors score accuracy at baseline and after each of three weeks. | Ching-Yi Huang, Yi-Pang Lee, Andy Sun, and Chun-Pin Chiang | Medicine | University | Not in cards | Jul 2025 | ||||
| TalentCLEF 2025 Multilingual Job Title Matching Models rank job titles by how closely they match a query title within or across languages. Organizers score mean average precision. | Avature and UNED | Other professions | Everyday | Not in cards | Sep 2024 | ||||
| TalentCLEF 2026 Contextualized Job-Person Matching Models rank candidate resumes for a job description in English or Spanish. Organizers score retrieval within and across languages and compare rankings across gender variants. | Avature, TechWolf, and UNED | Other professions | University | Not in cards | Nov 2025 | ||||
| TalentCLEF 2026 Job-Skill Matching with Skill Type Classification Models rank skills for an English job title. The main ranking metric gives core skills more weight than contextual skills. | Avature, TechWolf, and UNED | Other professions | University | Not in cards | Nov 2025 | ||||
| TalentCLEF 2025 Job Title-Based Skill Prediction Replaced in Nov 2025 | Avature and UNED | Not in cards | Sep 2024 | ||||||
| Talking Points: When Chatbots Surface Russian State Media Chatbots answer questions about Ukraine in five languages. Analysts check whether each response cites an official Russian channel, a state-controlled outlet, or a state-linked source. | Institute for Strategic Dialogue | Faithful to sources | Everyday | Not in cards | Oct 2025 | ||||
| TamperBench Open-weight models undergo fine-tuning and representation tampering attacks. Safety and utility metrics measure tamper resistance across attack and model pairs. | FAR.AI and Critical ML Lab | Jailbreaks and prompt injection | Expert | Not in cards | Feb 2026 | ||||
| TAP-Vid Models track individual points across frames and predict when each point disappears. The benchmark compares their trajectories and visibility predictions with reference tracks. | Google DeepMind | Video understanding | Everyday | Not in cards | Nov 2022 | ||||
| TarantuBench Agents complete simulated web security challenges. Each challenge has a binary success criterion. | Trivulzianus | Capture-the-flag challenges | Expert | Not in cards | Apr 2026 | ||||
| TartanAir v2 Models reconstruct camera motion and scene geometry from simulated trajectories. The evaluation compares predictions with known poses and depths. | TartanAir team | Spatial reasoning and counting | School | Not in cards | |||||
| Task AW IT support Agents complete five support workflows in simulated IT systems. Task checks cover onboarding, offboarding, and cleanup of accounts and assets. | abhishek203 | Customer service | University | Not in cards | Jul 2026 | ||||
| TAT-QA Models answer 16,552 questions about 2,757 passages from real financial reports that mix a table with text. Many answers need arithmetic, counting or comparison. Scored by exact match and F1. | National University of Singapore (NExT++) and collaborators | Finance | University | Not in cards | May 2021 | ||||
| Tatoeba Translation Challenge Models translate sentences for many language pairs. The challenge publishes held-out test sets so systems can report comparable translation scores. | Helsinki NLP | Translation | University | Not in cards | Nov 2020 | ||||
| TAU Urban Acoustic Scenes 2022 Mobile Models identify an urban acoustic scene from a one-second recording made on a mobile device. | Toni Heittola and colleagues | Audio understanding and voice | Everyday | Not in cards | Mar 2022 | ||||
| Tau2 Bench Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. | Snorkel AI | Customer service | University | Not in cards | Nov 2025 | ||||
| Tau2 Bench Verified Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. | Snorkel AI | Customer service | University | Not in cards | Dec 2025 | ||||
| Tavily Company Search Benchmark Search providers return evidence for company research questions, and an answer agent responds from those results. A judge scores answer completeness, relevance and freshness. | Tavily | Questions that need search | Expert | Not in cards | Oct 2026 | ||||
| TaxaBench-8k Models identify species or match observations across photographs, sounds, and other field data. The test contains 8,813 observations paired across six types of data. | Washington University in St. Louis | Genomics and life sciences | Expert | Not in cards | Oct 2024 | ||||
| TaxoBench Agents find research papers and organize them into a taxonomy. The evaluator compares their groupings and hierarchy with taxonomies from 72 published surveys. | Fudan University and Tencent Hunyuan | Deep research | Expert | Not in cards | Jan 2026 | ||||
| Te Hiku Media Māori speech recognition golden dataset Speech models transcribe te reo Māori recordings from native speakers and proficient second-language speakers. Te Hiku Media compares their word error rates. | Te Hiku Media | Speech recognition and translation | Everyday | Not in cards | Jan 2023 | ||||
| TE-ca Models decide whether a Catalan hypothesis follows from a premise. The reference labels identify the inference relation. | Barcelona Supercomputing Center | Reading in many languages | School | Not in cards | Jul 2021 | ||||
| TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. | Amazon Alexa AI | Simulated robots | Everyday | Not in cards | Oct 2021 | ||||
| Teaching Monster Challenge Agents generate instructional videos for a specified topic and learner. A model screens submissions, viewers compare videos in pairs, and an expert panel sets the final ranking. | National Taiwan University and collaborators | Other professions | University | Not in cards | Mar 2026 | ||||
| TED-LIUM Models transcribe English TED talks. The reference transcripts support comparisons by word error rate. | LIUM | Speech recognition and translation | Everyday | Not in cards | |||||
| Terminal-Bench Challenges Agents build a large project end to end with no time limit, such as a faster Rust compiler or a WebGL renderer in WebAssembly. Each challenge has its own test bed and metric. | Terminal-Bench team (Stanford, Harbor and Laude Institute) | Long terminal sessions | Research frontier | Not in cards | Jun 2026 | ||||
| TerraFirm REIT Models choose which properties to sell from a portfolio of 12 office buildings. The evaluator checks whether the remaining portfolio meets five debt and valuation constraints. | Patrick Browne | Finance | Expert | Not in cards | Feb 2026 | ||||
| TerretaQA Models answer open-domain questions in Valencian about the history, geography, tourism, and culture of the Valencian Community. | University of Alicante | Knowledge in many languages | School | Not in cards | Oct 2026 | ||||
| Testing Relational Understanding in Text-Guided Image Generation Models depict relationships between named entities. Human participants judge whether the generated scenes match each caption. | Colin Conwell and colleagues | Image generation and editing | Everyday | Not in cards | Jul 2022 | ||||
| Text2CAD Models generate parametric CAD designs from text prompts at different levels of detail. The authors compare generated geometry with the reference and count invalid designs. | Mohammad Sadil Khan and colleagues | CAD models | University | Not in cards | Sep 2024 | ||||
| TGIF-QA Models answer questions about actions, repetitions, and changes over time in animated clips. | Yunseok Jang and colleagues | Video understanding | Everyday | Not in cards | Apr 2017 | ||||
| The GenderQueer Test Suite Models translate 331 English passages into Icelandic. The graders check pronouns, adjective agreement and terms for LGBTQIA+ people. | University of Iceland | Translation | Everyday | Not in cards | Sep 2024 | ||||
| The Impossible Refund Assistants pursue a late train refund in a simulated email environment. Separate judges rate deception and other behaviors from the messages. | Vals AI | Deception and alignment | Everyday | Not in cards | Oct 2026 | ||||
| The Met Dataset Models match photographs of artworks to catalog images from the Metropolitan Museum of Art. The evaluation measures instance recognition across photographic conditions. | Nikolaos-Antonios Ypsilantis and colleagues | General visual questions | Everyday | Not in cards | |||||
| The Midnight Index Models lead a nation through fixed scenarios in a strategy game. The simulation records outcomes, escalation, and whether public commitments match later decisions. | Clouthier Labs | Deception and alignment | Everyday | Not in cards | Oct 2026 | ||||
| The Silicon Sample Benchmark Models predict how 16 messages affect trust in climate scientists and related outcomes. Organizers compare predictions with responses from a US experiment. | Jan Pfänder and the Silicon Sample Benchmark team | Social simulation | University | Not in cards | Jun 2026 | ||||
| The Turtle Test Models remember a conditional instruction during unrelated conversation. They must apply it when triggered and avoid applying it on earlier turns. | mohi_achu | Multi-turn conversation | Everyday | Not in cards | Oct 2026 | ||||
| TheGoldman Retrieval Models retrieve the Goldman Sachs dictionary definition for a financial term. The score rewards placing the correct definition near the top of the first ten results. | The Hong Kong University of Science and Technology | Finance | Everyday | Not in cards | Jul 2024 | ||||
| Third PASCAL Recognizing Textual Entailment Challenge Models decide whether a passage entails a proposed statement. The challenge compares their decisions with human labels. | Danilo Giampiccolo and colleagues | Reading comprehension | School | Not in cards | Jun 2007 | ||||
| TIME (time series forecasting) Models forecast time series over short, medium, and long horizons without training on the target task. | Real-TSF | Real-world events | University | Not in cards | Feb 2026 | ||||
| tinyBenchmarks Models answer small selected subsets of established benchmarks. Item response estimates approximate accuracy on the full tests. | Felipe Maia Polo and colleagues | Composite indices | University | Not in cards | Feb 2024 | ||||
| TituLLMs Bangla Evaluation Suite Models answer Bangla questions on academic subjects, science and everyday reasoning. The suite also asks yes-or-no questions about passages. | Hishab (TituLLMs team) | Knowledge in many languages | University | Not in cards | Feb 2025 | ||||
| TMHINT-QI version II v2 Models predict how clear Mandarin speech sounds and how many words listeners understand. The test adds street noise and two previously unseen enhancement systems. | Academia Sinica and VoiceMOS collaborators | Audio understanding and voice | Everyday | Not in cards | May 2023 | ||||
| TMHINT-QI v1 Replaced in May 2023 | Academia Sinica | Not in cards | Nov 2021 | ||||||
| TOEFL11 Essay Proficiency Scoring Models rate the English proficiency of each essay as low, medium, or high. Researchers measure agreement with the human ratings using quadratic weighted kappa. | Educational Testing Service | Other professions | University | Not in cards | Jan 2013 | ||||
| TokSuite Models complete questions under alternative tokenizations. The benchmark measures the change in accuracy relative to the canonical representation. | Gül Sena Altıntaş and colleagues | Reasoning in many languages | School | Not in cards | Dec 2025 | ||||
| ToLD-Br Models identify toxic language in Brazilian Portuguese posts using human annotations. | João Augusto Leite and colleagues | Bias, fairness and politics | Everyday | Not in cards | Oct 2020 | ||||
| Toloka Visual Question Answering Benchmark Models answer questions about images by locating the object that answers each question. The benchmark compares predicted and reference bounding boxes. | Toloka | Spatial reasoning and counting | Everyday | Not in cards | |||||
| ToolBench Models plan and call real REST APIs from RapidAPI, over 16,000 of them, to answer instructions that need one or many tools. ToolEval judges pass rate and win rate. | OpenBMB and Tsinghua University | Function calling | Everyday | Not in cards | Jul 2023 | ||||
| ToolRet Models retrieve relevant tools for 7,600 requests from a corpus of about 43,000 tools. Evaluation measures ranking quality, precision, recall, and complete tool-set retrieval. | Shandong University, Baidu, and Leiden University | Function calling | University | Not in cards | Mar 2025 | ||||
| TopiOCQA Models retrieve evidence and answer follow-up questions in conversations that change topics. | Vaibhav Adlakha and colleagues | Questions that need search | School | Not in cards | Oct 2021 | ||||
| ToxicChat Models identify toxic user messages from real conversations with an online chatbot. | Zi Lin and colleagues | Deception and alignment | School | Not in cards | Oct 2023 | ||||
| TraceSpatial-Bench Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth. | Beihang University + Peking University + BAAI | Embodied reasoning | Everyday | Not in cards | Dec 2025 | ||||
| Trading Bot Bench Models write Python trading strategies for historical stock prices. The evaluator combines return, Sharpe ratio, and drawdown on held-out scenarios. | amrita | Finance | University | Not in cards | Sep 2025 | ||||
| Transformer Puzzles (RASPy) Models complete eight sequence-programming puzzles in RASPy. The hub allows up to eight attempts per puzzle and checks solutions with executable tests. | Sasha Rush | Function-level problems | University | Not in cards | Nov 2022 | ||||
| Transluce Mental Health Evaluation Models hold multi-turn conversations with simulated users facing mental-health scenarios. Automated judges score 14 helpful and harmful behaviors. | Transluce | Wellbeing and sycophancy | Expert | Not in cards | Aug 2026 | ||||
| Transluce User Awareness Evaluation Models answer behavior, capability, grading and borderline-request tasks under different simulated user identities. Scores measure identity-conditioned behavior shifts. | Transluce | Safety and alignment | Expert | Not in cards | Aug 2026 | ||||
| TravelPlanner Agents plan multi-day trips from a user request, using tools to search nearly four million records of flights, hotels, restaurants and attractions. Each plan must meet the user's hard constraints, such as budget, and commonsense rules. The headline final pass rate counts a plan only when it meets every constraint. | Ohio State University (OSU NLP Group) and Fudan University | General assistants | Everyday | Not in cards | Feb 2024 | ||||
| TREC 2017 LiveQA Medical Task Models answer consumer health questions. The shared task compares responses with expert assessments of answer quality. | Asma Ben Abacha and colleagues | Medicine | Expert | Not in cards | |||||
| TREC 2023 Clinical Trials Track Models rank trials for 50 simulated patients using answers to medical intake questionnaires. Some answers can be missing. | TREC Clinical Trials Track organizers | Medicine | Expert | Not in cards | May 2023 | ||||
| TREC Million LLMs 2025 Systems rank simulated expert language models for a new question using their answers to earlier questions. Each expert has access to a different document cluster. | NIST TREC Million LLMs Track | Questions that need search | University | Not in cards | Nov 2024 | ||||
| TREC Product Search 2025 Models rewrite shopping queries for a search engine. Automatic entries submit one reformulation. Interactive entries can submit four, and the best one counts. | NIST TREC Product Search and Recommendation Track | Questions that need search | Everyday | Not in cards | Mar 2025 | ||||
| TREC Product Search 2024 Replaced in Mar 2025 | NIST TREC Product Search Track | Not in cards | Jan 2024 | ||||||
| TREC Product Search 2023 Replaced in Jan 2024 | NIST TREC Product Search Track | Not in cards | Dec 2022 | ||||||
| TREC Related Product Recommendation 2025 Models rank items that complement or substitute for a given product. Assessors label the type and strength of each relationship. | NIST TREC Product Search and Recommendation Track | Questions that need search | Everyday | Not in cards | Mar 2025 | ||||
| TREC-COVID Models rank scientific articles that address specific information needs about COVID-19. | National Institute of Standards and Technology and collaborators | Questions that need search | Expert | Not in cards | Apr 2020 | ||||
| TREK-150 Models follow a marked object through first-person kitchen videos. In the one-pass test, they receive its bounding box in the first frame and track it to the end. | University of Udine and University of Catania | Video understanding | Everyday | Not in cards | Aug 2021 | ||||
| Trialstreamer clinical evidence extraction Models identify treatments, comparators, outcomes, and the direction of each reported effect in trial abstracts. Physicians annotated 100 test abstracts. | Northeastern University and collaborators | Medicine | University | Not in cards | Jan 2020 | ||||
| TriScore Models identify composers and instruments from recordings, score images, and symbolic notation. They also retrieve excerpts from the same composition. | University of Alicante | Audio understanding and voice | University | Not in cards | Apr 2026 | ||||
| Trivia-et Models answer Estonian trivia questions from the Eesti Mäng board game. | TalTechNLP | Knowledge in many languages | Everyday | Not in cards | Dec 2024 | ||||
| TriWorldBench World models generate synchronized robot-view videos from head and wrist cameras. Nineteen signals measure cross-view agreement, task alignment, physical coherence, motion, temporal stability, and visual quality. | Peking University, Tsinghua University, Beihang University and others | Video world models | Everyday | Not in cards | Sep 2026 | ||||
| TruthInsightBench Agents analyze data from published studies without seeing the original conclusions. A judge grades whether the evidence and artifacts support the claims each agent makes. | TruthInsight-AI | Research science | Expert | Not in cards | Sep 2026 | ||||
| Turkish Dental Specialty Examination 2020 and 2021 Models answer Turkish dental specialty entrance questions in basic and clinical sciences. The two official answer keys cover 238 scored questions, including questions with images. | ÖSYM | Medicine | Expert | Not in cards | Jan 2025 | ||||
| Turku NER Corpus Models identify named entities in Finnish text from the Universal Dependencies corpus. | Jouni Luoma and colleagues | Reading in many languages | School | Not in cards | May 2020 | ||||
| Turku Paraphrase Corpus Models compare Finnish sentence pairs and estimate how closely their meanings match. | Jenna Kanerva and colleagues | Reading in many languages | School | Not in cards | May 2021 | ||||
| TUT Urban Acoustic Scenes 2018 Models recognize the setting of a ten-second audio clip recorded in a European city. | Annamaria Mesaros and colleagues | Audio understanding and voice | Everyday | Not in cards | Apr 2018 | ||||
| TV-AD Models describe what viewers can see during clips from television episodes. The test pairs each clip with a timed reference description. | University of Oxford, École des Ponts and Shanghai Jiao Tong University | Video captioning | Everyday | Not in cards | Jul 2024 | ||||
| tweetSentBR Models classify the sentiment of Brazilian Portuguese social media posts as positive, neutral, or negative. | Henrique Brum and Maria das Graças Volpe Nunes | Reading in many languages | Everyday | Not in cards | May 2018 | ||||
| Twitter Financial News Sentiment Models classify financial news posts as bearish, bullish, or neutral. The released validation file contains 2,388 posts. | zeroshot | Finance | Everyday | Not in cards | Sep 2022 | ||||
| Twitter URL Paraphrase Corpus Models identify paraphrases among pairs of social media posts that reference the same URL. | Wuwei Lan and colleagues | Reading comprehension | School | Not in cards | Sep 2017 | ||||
| Two-Hop Latent Reasoning Models answer questions that require composing two facts. The hub port covers the semi-synthetic tasks and includes a shuffled control. | Mikuláš Balesni, Tomasz Korbak and Owain Evans | Language reasoning | University | Not in cards | Nov 2024 | ||||
| Typed Decision Bench Models assign probabilities to decisions about a given state. The benchmark scores accuracy and calibration across five suites. | Blobfish AI | Running a business | Expert | Not in cards | Sep 2026 | ||||
| UAV Operator Models manage simulated drone missions as weather and flight restrictions change. The simulator scores their decisions from logged aircraft and mission state. | jarrett | Simulated robots | Expert | Not in cards | Jul 2026 | ||||
| UAV Robust Control Hub Agents construct policies for a fictional civilian drone simulator. The benchmark evaluates policies on hidden scenarios and reports separate results under changed conditions. | andehit | Simulated robots | University | Not in cards | Nov 2026 | ||||
| UCF101 Models recognize one of 101 human actions in video clips. | University of Central Florida | Video understanding | Everyday | Not in cards | Dec 2012 | ||||
| UCI Adult Dataset (Project Moonshot) Models choose an income range or say that income cannot be determined from a census profile. Moonshot measures exact matches with the expected response. | AI Verify Foundation | Bias, fairness and politics | Everyday | Not in cards | May 2024 | ||||
| UCM-Captions Models describe aerial land use scenes from the UC Merced collection. The dataset pairs 2,100 images with five human captions each. | Bo Qu, Xuelong Li, Dacheng Tao, and Xiaoqiang Lu | Detection and captioning | Everyday | Not in cards | Jul 2016 | ||||
| UD Ancient Greek Perseus Models identify dictionary forms, parts of speech, and grammatical relationships in Ancient Greek. The version 2.10 test contains 1,306 sentences from Athenaeus. | Tufts University and Leipzig University | Reading in many languages | University | Not in cards | Nov 2015 | ||||
| UD Ancient Greek PROIEL Models analyze Ancient Greek passages from Herodotus and the New Testament. They identify dictionary forms and grammatical relationships between words. | University of Oslo | Reading in many languages | University | Not in cards | Nov 2015 | ||||
| UD Ancient Greek PTNK Models tag and parse the Greek Septuagint. The test contains 410 verses from chapters 19 through 30 of Genesis. | Daniel Swanson, Bryce Bussert, and Francis Tyers | Reading in many languages | University | Not in cards | Nov 2023 | ||||
| UD Ancient Hebrew PTNK Models analyze word forms and grammatical relationships in the Hebrew Bible. Version 2.17 tests 410 verses from Genesis and 142 from Deuteronomy. | Daniel Swanson and Francis Tyers | Reading in many languages | University | Not in cards | May 2022 | ||||
| UD Latin CIRCSE Models tag and parse Latin drama and prose. The version 2.14 test contains two tragedies by Seneca and Germania by Tacitus. | CIRCSE Research Centre, Università Cattolica del Sacro Cuore | Reading in many languages | University | Not in cards | May 2024 | ||||
| UD Latin ITTB Models analyze medieval Latin prose by Thomas Aquinas. They identify the grammatical form of each word and its relationship to other words. | Index Thomisticus Treebank team | Reading in many languages | University | Not in cards | Nov 2015 | ||||
| UD Latin LLCT Models tag and parse the Latin of legal charters from Tuscany. The source documents date from 774 to 897. | Timo Korkiakangas, Flavio Massimiliano Cecchini, and Marco Passarotti | Reading in many languages | University | Not in cards | May 2020 | ||||
| UD Latin Perseus Models analyze word forms and grammatical relationships in Latin literary texts. Version 2.13 reserves 939 sentences across four source files for testing. | Tufts University and Leipzig University | Reading in many languages | University | Not in cards | Nov 2015 | ||||
| UD Latin PROIEL Models analyze Latin sentences from the Vulgate and from authors including Caesar and Cicero. The test includes both biblical and secular prose. | University of Oslo | Reading in many languages | University | Not in cards | Nov 2015 | ||||
| UD Latin UDante Models tag and parse Dante's Latin. All 419 test sentences in version 2.13 come from his treatise on the vernacular, De vulgari eloquentia. | University of Pisa and Università Cattolica del Sacro Cuore | Reading in many languages | University | Not in cards | May 2021 | ||||
| Uganda Cultural and Cognitive Benchmark Models answer questions about Ugandan daily life, institutions and cultural practices. A grader checks answers across 24 domains. | Crane AI Labs | Trivia and open questions | Everyday | Not in cards | Jul 2025 | ||||
| UI-Vision Computer use agents are tested offline on human desktop demonstrations from 83 applications. Three tasks go from element grounding to layout grounding to next-action prediction for clicks, drags and key presses. | ServiceNow Research and collaborators | Finding things on screen | Everyday | Not in cards | Mar 2025 | ||||
| UI-Vision Action Prediction Models predict the next desktop interface action. The score is the published step success rate. | ServiceNow Research and collaborators | Finding things on screen | Everyday | Not in cards | Mar 2025 | ||||
| UIT-ViCTSD Models classify Vietnamese comments for constructiveness and toxicity. The two labels capture different properties of the same text. | Son T. Luu and colleagues | Reading in many languages | Everyday | Not in cards | Mar 2021 | ||||
| UIT-ViHSD Models classify Vietnamese social media text into three categories covering clean language, offensive language, and hate speech. | Son T. Luu, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen | Reading in many languages | Everyday | Not in cards | |||||
| UIT-VSFC Models classify sentiment in Vietnamese student feedback. The corpus also includes topic labels. | Kiet Van Nguyen and colleagues | Reading in many languages | Everyday | Not in cards | Nov 2018 | ||||
| UK Dermatology SCE Sample Questions Models answer multiple-choice questions from the Federation’s dermatology practice exam. Researchers score answers against the official key. | Federation of the Royal Colleges of Physicians of the UK | Medicine | Expert | Not in cards | Jun 2023 | ||||
| ULSA synthesis action retrieval Models identify and classify actions in sentences about inorganic materials synthesis. The corpus contains 3,040 labeled sentences from 535 paragraphs. | University of California, Berkeley and Lawrence Berkeley National Laboratory | Lab and data work | University | Not in cards | Oct 2021 | ||||
| Unhealthy Comments Corpus Models rate newspaper comments for overall conversational health and six attributes, including hostility and sarcasm. Researchers score each label separately using ROC AUC. | Ilan Price and colleagues | Reading comprehension | Everyday | Not in cards | Aug 2020 | ||||
| Unidentified Video Objects Models segment and track objects in video without a fixed list of object categories. The evaluation compares predicted masks and tracks with annotated instances. | UVO team | Video understanding | Everyday | Not in cards | Apr 2021 | ||||
| Unit Commitment Lite Models schedule generators to serve hourly demand at minimum cost. The grader checks whether schedules meet operating constraints and compares cost with the optimum. | jwilksbooth | Electronics | Expert | Not in cards | Jul 2026 | ||||
| Universal Dependencies Sanskrit Vedic Models identify which word governs each word in Vedic Sanskrit and label the grammatical relation. The official split keeps each hymn or prose chapter together. | University of Zurich and Universal Dependencies contributors | Reading in many languages | University | Not in cards | May 2020 | ||||
| UNT Computer Science Short Answer Dataset 2.0 Models grade 2,273 student answers from a computer science course against the average of two human grades. The authors report root mean square error and correlation. | University of North Texas | Other professions | University | Not in cards | Apr 2011 | ||||
| UQ Models answer 500 unsolved Stack Exchange questions. A model checks answers without a reference solution, and community members review selected answers. | UQ project | Expert-written questions | Research frontier | Not in cards | Aug 2025 | ||||
| UrbanSED Models locate sound events in synthetic urban soundscapes. The benchmark includes reference labels for the times and types of events. | Justin Salamon and colleagues | Audio understanding and voice | Everyday | Not in cards | Oct 2017 | ||||
| Urdu Language Speech Emotional Corpus Models identify emotions in excerpts from Urdu television talk shows. The recordings cover 38 speakers and four emotion categories. | Siddique Latif, Adnan Qayyum, Muhammad Usman and Junaid Qadir | Audio understanding and voice | Everyday | Not in cards | Jun 2018 | ||||
| UrduSER Models identify emotions in Urdu dialogue from Pakistani television. The corpus contains 3,500 excerpts from ten actors across seven emotion categories. | Muhammad Zaheer Akhtar and colleagues | Audio understanding and voice | Everyday | Not in cards | Dec 2024 | ||||
| URGENT 2024 Models enhance speech under different forms of degradation. NVIDIA measures how accurately a recognizer transcribes the evaluation audio. | URGENT Challenge organizers | Speech recognition and translation | Everyday | Not in cards | |||||
| US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. | Medicine | Expert | Not in cards | Jul 2025 | |||||
| UW Clinical Record Section Chunker i2b2 Discharge Summary Corpus Models locate section boundaries in discharge summaries and label each section. Researchers compare the predicted labels and boundaries with human annotations. | University of Washington and Microsoft Research | Medicine | University | Not in cards | May 2012 | ||||
| UWBench Models answer questions about underwater scenes, write captions, and locate objects from descriptions. Questions cover visibility, materials, and spatial relationships. | Institute of Artificial Intelligence, China Telecom | General visual questions | Everyday | Not in cards | Oct 2025 | ||||
| Uyghur Language Question Answering Models answer crowdsourced Uyghur language and literature questions. The suite includes reading comprehension, language exercises and exam questions at different proficiency levels. | keramjan (Uyghur evaluation datasets) | Reading in many languages | School | Not in cards | Jan 2026 | ||||
| Uzbek Open ASR Leaderboard Models transcribe Uzbek recordings from four test sets. The main score averages the four word error rates with equal weight. | NavAI | Speech recognition and translation | Everyday | Not in cards | Jul 2026 | ||||
| Uzbek Speech Corpus Models transcribe held-out recordings of people reading Uzbek. Scorers compare the transcripts with the reference words. | ISSAI and Tashkent University of Information Technologies | Speech recognition and translation | Everyday | Not in cards | Jul 2021 | ||||
| V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. | Meta | Real robots | Everyday | Not in cards | Jun 2025 | ||||
| Vaani Benchmark V1.0 Models transcribe speakers describing pictures. The scorer compares each prediction with three human transcripts and allows wording that matches any reference. | ARTPARK and Indian Institute of Science | Speech recognition and translation | Everyday | Not in cards | Jun 2026 | ||||
| VABAR Agents program robots to sort tools, mesh gears, and handle cables in simulation. The harness checks success as object shapes and positions vary. | UC Berkeley, NVIDIA, and collaborators | Simulated robots | University | Not in cards | Nov 2026 | ||||
| VAERS influenza vaccine adverse event extraction Models mark six kinds of clinical information in influenza vaccine safety reports, including neurological events, procedures, and time expressions. Researchers score exact or overlapping text spans. | University of Texas Health Science Center at Houston and University of Pennsylvania | Medicine | University | Not in cards | Aug 2020 | ||||
| Valerius Flaccus Intertextuality Benchmark Models find passages in four Latin epics that scholars linked to Book 1 of the Argonautica. The authors score precision and recall against 945 parallels. | Quantitative Criticism Lab | Reading in many languages | Expert | Not in cards | May 2021 | ||||
| VALOR-32K Models match captions to video and audio clips. Meta reports separate retrieval results for text paired with audio and text paired with video. | Jing Liu, Sihan Chen, and colleagues | Audio understanding and voice | Everyday | Not in cards | Apr 2023 | ||||
| Vals Legal AI Report Legal AI tools perform seven kinds of legal work. Vals compares their task scores with scores from practicing lawyers. | Vals AI | Law | Expert | Not in cards | Feb 2025 | ||||
| Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. | Vals AI | Composite indices | Expert | Not in cards | Oct 2026 | ||||
| Vals Smith: pingdotgg/t3code Agents modify pingdotgg/t3code to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Vals AI | Fixing real repositories | Expert | Not in cards | Nov 2026 | ||||
| VCB-Bench Audio language models answer spoken prompts across instruction following, empathy, safety, simulation and vocalization tasks. Judges score the responses. | Tencent | Audio understanding and voice | Expert | Not in cards | Oct 2025 | ||||
| VCTK Corpus Models match speech recordings with their texts. Meta evaluates audio-to-text retrieval across model sizes. | Centre for Speech Technology Research at the University of Edinburgh | Audio understanding and voice | Everyday | Not in cards | |||||
| Verifiability Judgments for Generative Search Models decide whether a cited source fully, partly, or never supports a generated claim. Human annotations provide the reference judgments. | Nelson F. Liu and colleagues | Faithful to sources | School | Not in cards | Apr 2023 | ||||
| Verifiable Browsing Task (Meta) Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools. | Meta (Meta Superintelligence Labs) | Web browsing | Everyday | Not in cards | Oct 2026 | ||||
| VerticalRTK Search and answer APIs answer questions about facts that change over time across seven real-world domains. An LLM judge compares answers with dated references. | Tako | Deep research | Expert | Not in cards | Jul 2026 | ||||
| VeruSAGE-Bench repository Agents synthesize Verus proofs for 849 tasks extracted from eight real-world Rust systems projects. | Microsoft Research and collaborators | Fixing real repositories | Research frontier | Not in cards | Dec 2025 | ||||
| Verus-Bench algorithm Replaced in Dec 2025 | Function-level problems | Expert | Not in cards | Oct 2024 | |||||
| VGGSound Models identify sound events in video clips where the sound source is visible. | University of Oxford | Audio understanding and voice | Everyday | Not in cards | Apr 2020 | ||||
| Video Browser Showdown 2026 marine video search Participants use interactive retrieval systems to find five underwater clips after seeing each target. Each search has a five-minute limit. | Video Browser Showdown organizers | Video understanding | Everyday | Not in cards | Jan 2026 | ||||
| Video Object Segmentation under Transformations Models follow objects whose appearance changes as people manipulate them. The benchmark compares predicted masks with annotations through these transformations. | VOST team | Video understanding | Everyday | Not in cards | Dec 2022 | ||||
| Video to Valence Models predict which of two videos viewers find more pleasant. Human preference annotations supply the reference judgments. | Mantas Mazeika and collaborators | Video understanding | Everyday | Not in cards | Oct 2022 | ||||
| Video-ChatGPT video conversation benchmark Models answer questions about video content. The evaluation grades correctness, detail, context, timing, and consistency. | Muhammad Maaz and colleagues | Video captioning | Everyday | Not in cards | Jun 2023 | ||||
| Video-MonetBench Models rank two videos generated for the same prompt. The test compares each ranking with a human preference, including ties. | Tsinghua University and Z.ai | Video understanding | Everyday | Not in cards | Dec 2024 | ||||
| Video2Robot Embodied Models predict cleaned two-dimensional trajectories from noisy or occluded landmark tracks. The evaluator compares predicted motion and hand grips with a hidden synthetic reference. | taylormino | Embodied reasoning | University | Not in cards | Oct 2026 | ||||
| VideoCon Models distinguish descriptions supported by a video from captions that contradict it. | Hritik Bansal and colleagues | Video understanding | School | Not in cards | Nov 2023 | ||||
| VideoGen-RewardBench Models rank two generated videos for the same prompt. The scorer compares their rankings with human preferences for visual quality, motion, prompt alignment, and overall quality. | Kuaishou Technology and collaborators | Video understanding | Everyday | Not in cards | Jan 2025 | ||||
| ViGiL3D Models locate objects in indoor 3D scans from English descriptions. A description can match several objects or none. Models must return the matching set. | Simon Fraser University and Alberta Machine Intelligence Institute | Spatial reasoning and counting | Everyday | Not in cards | Jan 2025 | ||||
| ViLegalExpert Models retrieve Vietnamese law and answer questions from citizen consultations. Professional answers and expert-verified provisions supply references across 34 legal domains. | University of Information Technology, Vietnam National University Ho Chi Minh City | Law | University | Not in cards | Sep 2026 | ||||
| ViLLM-Eval Models answer Vietnamese exam, general knowledge, and reading comprehension questions. They also predict the final word in a passage. | Ton Duc Thang University, INT2 and HPC SYSTEMS | Knowledge in many languages | School | Not in cards | Apr 2024 | ||||
| VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. | Stanford University and NVIDIA with collaborators | Simulated robots | Everyday | Not in cards | Oct 2022 | ||||
| Vinoground Models match short videos with captions that distinguish events by their temporal order. | Jianrui Zhang and colleagues | Video understanding | School | Not in cards | Oct 2024 | ||||
| ViNV-TTS Models read Vietnamese dialogue aloud and place sounds such as laughter or breathing where the text marks them. | VLSP 2026 organizers | Audio understanding and voice | Everyday | Not in cards | Sep 2026 | ||||
| VIPBench Models predict whether listeners hear the same voice in two recordings. The authors compare eleven speech models against judgments from 1,290 listeners. | Northwestern University | Speech recognition and translation | Everyday | Not in cards | Nov 2026 | ||||
| VIPSeg Models label video pixels and keep instance identities consistent across frames. The benchmark includes both countable objects and background regions. | VIPSeg team | Video understanding | Everyday | Not in cards | |||||
| Virtual KITTI 2 Models interpret simulated driving videos with reference geometry and pixel labels. Meta uses the sequences to compare video object segmentation models. | Naver Labs Europe | Video understanding | Everyday | Not in cards | Jan 2020 | ||||
| VisR-RAG-Fin Models retrieve pages from financial documents and answer questions using their contents. The evaluator measures page ranking and checks answers against references. | University of Luxembourg and collaborators | Finance | University | Not in cards | Oct 2026 | ||||
| Vistaar Models transcribe speech across 59 combinations of language and dataset in 12 Indian languages. Each test has its own word error rate. | AI4Bharat | Speech recognition and translation | Everyday | Not in cards | May 2023 | ||||
| VizWiz Models answer over 31,000 spoken questions that blind people asked about photos they took with phones. Some questions are unanswerable from the image. | University of Texas at Austin and collaborators (Gurari et al.) | General visual questions | Everyday | Not in cards | Feb 2018 | ||||
| VLAIR Legal Research Legal AI tools answer 200 US legal research questions. Reviewers score legal accuracy, citation authority, and suitability for sharing. | Vals AI | Law | Expert | Not in cards | Oct 2025 | ||||
| VLSP 2023 Vietnamese Mispronunciation Detection Models compare spoken Vietnamese with the expected pronunciation and flag individual sounds that the speaker pronounced incorrectly. | VLSP 2023 organizers | Speech recognition and translation | Everyday | Not in cards | Aug 2023 | ||||
| VM14K Models answer Vietnamese multiple-choice questions across 34 medical specialties. The authors reserve a private set for leaderboard evaluation. | Thong Nguyen and colleagues | Medicine | University | Not in cards | May 2025 | ||||
| Vmax Tasks Agents fix JavaScript bugs from projects including Vue.js, Redux, Docusaurus, and Chalk. Harbor lists 1,043 tasks with bug reports and expected behavior. | VmaxAI | Fixing real repositories | Expert | Not in cards | Jan 2026 | ||||
| VocalSound Models classify human vocal sounds such as laughter, coughing, and sneezing. | Yuan Gong and colleagues | Audio understanding and voice | Everyday | Not in cards | May 2022 | ||||
| Voice Agent LLM Benchmark Models handle simulated caller requests in text. Hamming grades task completion, policy compliance, and how naturally the models phrase their replies. | Hamming AI | Customer service | Everyday | Not in cards | Nov 2026 | ||||
| Voice Jailbreak Attacks Against GPT-4o Models respond to spoken requests. Human reviewers measure how often a model complies with an unsafe request. | Xinyue Shen and colleagues | Jailbreaks and prompt injection | Everyday | Not in cards | May 2024 | ||||
| VoiceMOS Challenge 2026 Models predict listener ratings for enhanced speech, emotional speech, and synthesized accents. Organizers compare predictions with ratings for individual utterances. | VoiceMOS Challenge organizing committee | Speech recognition and translation | Everyday | Not in cards | Apr 2026 | ||||
| AudioMOS Challenge 2025 Replaced in Apr 2026 | AudioMOS steering group, Nagoya University, Nankai University, Meta AI, and NICT | Audio understanding and voice | Not in cards | Mar 2025 | |||||
| VoiceMOS Challenge 2024 Replaced in Mar 2025 | Audio understanding and voice | Not in cards | Apr 2024 | ||||||
| VoiceMOS Challenge 2023 Replaced in Apr 2024 | Audio understanding and voice | Not in cards | Apr 2023 | ||||||
| VoxAnon Models anonymize speech or try to identify the original speaker. Balanced trials compare reidentification across age, gender, pitch, and accent groups. | Avignon University | Speech recognition and translation | Everyday | Not in cards | Oct 2026 | ||||
| VoxCeleb1 Models identify speakers from recorded speech. NVIDIA also uses the recordings for speaker counting within AIR-Bench. | University of Oxford | Audio understanding and voice | Everyday | Not in cards | Jun 2017 | ||||
| VoxLingua107 Models identify the language in a speech recording. The corpus includes 107 languages. | Tallinn University of Technology | Audio understanding and voice | Everyday | Not in cards | Nov 2020 | ||||
| VoxPopuli Models transcribe European Parliament speeches. The corpus also supports speech translation across languages. | Facebook AI | Speech recognition and translation | Everyday | Not in cards | Jan 2021 | ||||
| VQA-RAD Vision-language models answer clinicians' questions about 315 radiology images, including head CTs, MRIs and chest X-rays. Questions are closed yes or no, or open free text. | NIH and collaborators (Lau et al.) | Medicine | Expert | Not in cards | Nov 2018 | ||||
| VSAS-Bench Streaming vision-language models answer tasks over online video frames while evaluation measures accuracy, response timing and consistency. | Apple | Video understanding | Expert | Not in cards | Apr 2026 | ||||
| VSEC Models correct misspellings in Vietnamese text. The evaluation compares their corrections against reference sentences. | Dinh-Truong Do and colleagues | Reading in many languages | School | Not in cards | Nov 2021 | ||||
| VSI-SUPER Video models watch arbitrarily long streams and must recall where objects appeared (VSR) or keep counting objects across scenes (VSC). Video lengths run from 10 minutes to 4 hours. | New York University (Cambrian-S team, Yang et al.) | Long videos | Everyday | Not in cards | Nov 2025 | ||||
| VSR Models judge whether captions with 65 kinds of spatial relation, such as under or facing, are true of an image. Over 10,000 text and image pairs. | University of Cambridge (Liu et al.) | Spatial reasoning and counting | Everyday | Not in cards | Apr 2022 | ||||
| VSR Random Split Models judge whether image captions correctly describe spatial relations on the randomly partitioned test set. The score is accuracy over 2,195 test examples. | University of Cambridge (Liu et al.) | Spatial reasoning and counting | Everyday | Not in cards | Apr 2022 | ||||
| VSR Zero-Shot Split Models judge spatial captions on a test set whose object concepts do not appear in training. The score is accuracy over 616 test examples. | University of Cambridge (Liu et al.) | Spatial reasoning and counting | Everyday | Not in cards | Apr 2022 | ||||
| VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. | Tencent | Finding and exploiting real bugs | Expert | Not in cards | Apr 2026 | ||||
| Wahl-O-Mat 2024 European election party agreement Models take a position on 38 German statements about European Union policy. Researchers compare those answers with each party’s published positions. | Luca Rettenberger, Markus Reischl, and Mark Schutera at Karlsruhe Institute of Technology | Bias, fairness and politics | Everyday | Not in cards | May 2024 | ||||
| Wahl-O-Mat 2025 Bundestag party agreement Models answer 38 statements from Germany’s 2025 federal election. The original researchers score agreement with each of seven parties using the Wahl-O-Mat weighting rule. | Ina Dormuth and colleagues at TU Dortmund University | Bias, fairness and politics | Everyday | Not in cards | Feb 2025 | ||||
| WANLP 2022 Shared Task on Propaganda Detection in Arabic Models identify persuasion techniques in Arabic social media posts. A second task asks for the words that express each technique. | WANLP 2022 shared task organizers | Reading in many languages | University | Not in cards | Jul 2022 | ||||
| Washington Post midterm voter guides Chatbots write voter guides for US addresses. The Washington Post checks whether each guide names the correct House district. | The Washington Post and Promptwatch | Long-form factuality | Everyday | Not in cards | Oct 2026 | ||||
| WASSCE Benchmark for Open Language Models Models answer questions based on the West African Senior School Certificate Examination. The harness compares models across mathematics, English, science, and social studies. | acheamponge | Exam questions | School | Not in cards | Apr 2026 | ||||
| WAXAL Speech Recognition Test Speech models transcribe people describing pictures in African languages. Researchers compare the transcripts with human references using word error rate. | Google Research and African data partners | Speech recognition and translation | Everyday | Not in cards | Jan 2026 | ||||
| Wayfinder Code Search Benchmark Coding search agents find the files changed by real bug fixes from short commit subjects or longer problem briefs. | Voyage AI | Fixing real repositories | Expert | Not in cards | Nov 2026 | ||||
| WebGuard Models classify the risk of a web action using human labels for safe, low-risk, and high-risk actions. | Ohio State University and collaborators with Scale AI | Web browsing | Everyday | Not in cards | Jul 2025 | ||||
| WebShop Agents search and click through a simulated shopping site with 1.18 million real products to buy an item that matches a text instruction. Reward reflects how well the purchase matches. | Princeton University | Web browsing | Everyday | Not in cards | Jul 2022 | ||||
| WebSRC Models answer questions about a web page from its screenshot, HTML or both. The set has 400K question and answer pairs from 6.4K web pages. Scored by exact match and F1. | Shanghai Jiao Tong University (X-LANCE lab) | Documents, charts and text in images | Everyday | Not in cards | Jan 2021 | ||||
| WeChat Pay Physical Client Evaluation Agents operate the WeChat Pay client on physical phones. The team scores execution paths, use of supplied knowledge, acceptance checks, and account creation accuracy. | Tencent WeChat Pay | Phones | University | Not in cards | Nov 2026 | ||||
| WeirdChat Models respond to prompts designed to elicit unexpected behaviors. A judge labels more than 1,300 behavior patterns across annotated transcripts. | Transluce | Safety and alignment | Expert | Not in cards | Jul 2026 | ||||
| WenetSpeech Test_Meeting Models transcribe spontaneous Mandarin speech from real meetings in the Test_Meeting set. The baseline score is mixture error rate, counting Mandarin characters and English words. | Northwestern Polytechnical University (ASLP@NPU), Mobvoi and others | Speech recognition and translation | Everyday | Not in cards | Oct 2021 | ||||
| WenetSpeech Test_Net Models transcribe Mandarin internet speech from the Test_Net set. The baseline score is mixture error rate, counting Mandarin characters and English words. | Northwestern Polytechnical University (ASLP@NPU), Mobvoi and others | Speech recognition and translation | Everyday | Not in cards | Oct 2021 | ||||
| WenetSpeech-Chuan-Eval Models transcribe Sichuanese speech. Qwen reports separate error rates on the easy and hard test sets. | Yuhang Dai and colleagues | Speech recognition and translation | Everyday | Not in cards | Sep 2025 | ||||
| WHAMR! Models recover individual voices from noisy speech mixtures with simulated room reverberation. The test contains 3,000 mixtures. | Mitsubishi Electric Research Laboratories and Whisper | Audio understanding and voice | University | Not in cards | Oct 2019 | ||||
| WHAM! Replaced in Oct 2019 | Not in cards | Jul 2019 | |||||||
| Where Did My Data Go? Models trace missing records from a data source to a dashboard. The grader checks whether they find the cause or recognize that the evidence is insufficient. | Olga Braginskaya | Data science and SQL | University | Not in cards | Oct 2026 | ||||
| WHU-CDC Models compare aerial photos of Christchurch, describe what changed, and mark changed pixels. Reference captions describe reconstruction after the earthquake. | Jingye Shi and colleagues | Detection and captioning | Everyday | Not in cards | Oct 2024 | ||||
| Widget Captioning Models describe an individual interface element in a mobile screenshot. The benchmark compares their captions with human descriptions of the same element. | Google Research | Documents, charts and text in images | School | Not in cards | Oct 2020 | ||||
| Wiki Neutrality Corpus Models identify biased wording in English Wikipedia sentences and rewrite the sentences. The authors compare the rewrites with corrections by Wikipedia editors. | Reid Pryzant and colleagues | Bias, fairness and politics | School | Not in cards | Nov 2019 | ||||
| WIKIBIAS Models classify subjective bias in Wikipedia sentences, mark the words that express it, and rewrite the sentences. The test includes edits that change several separate phrases. | Yang Zhong, Jingfeng Yang, Wei Xu, and Diyi Yang | Bias, fairness and politics | School | Not in cards | Oct 2021 | ||||
| WikiCompany2IndustryClustering Models group company descriptions by industry. The score measures how well the groups recover the industry labels. | The Hong Kong University of Science and Technology | Finance | Everyday | Not in cards | Jul 2024 | ||||
| WikiFact (HELM) Models complete a Wikidata relation given its subject and predicate. HELM checks the predicted entity against a reference answer. | Stanford CRFM | Trivia and open questions | School | Not in cards | Nov 2022 | ||||
| WikiLingua Models summarize procedural articles in different languages. Aligned WikiHow articles support both same-language and cross-language evaluation. | Faisal Ladhak and colleagues | Reading in many languages | School | Not in cards | Nov 2020 | ||||
| WikiNEuRal Models locate named entities in Wikipedia text across nine languages using automatically produced reference annotations. | Simone Tedeschi and colleagues | Reading in many languages | School | Not in cards | Nov 2021 | ||||
| Wikispeedia Players navigate from one Wikipedia article to another using only article links. The Prime Intellect port scores whether agents reach targets held out from training. | Robert West, Joelle Pineau, and Doina Precup | Web browsing | Everyday | Not in cards | Jul 2009 | ||||
| WikiSQL Models turn natural language questions into SQL queries over single Wikipedia tables. The dataset has 80,654 hand-annotated questions over 24,241 tables. Scored by execution accuracy and logical form accuracy. | Salesforce Research (Zhong, Xiong, Socher) | Data science and SQL | School | Not in cards | Aug 2017 | ||||
| WikiTableQuestions Models answer complex questions about semi-structured HTML tables from Wikipedia. Answers often need counting, comparison or aggregation across rows. Scored by answer accuracy against the gold answers. | Stanford University (Pasupat and Liang) | Language reasoning | Everyday | Not in cards | Aug 2015 | ||||
| WikiText-2 Language models predict the next token in held-out Wikipedia articles. Lower perplexity means the model gave more probability to the tokens that actually appeared. | Stephen Merity, Caiming Xiong, James Bradbury and Richard Socher | Other | University | Not in cards | Sep 2016 | ||||
| WikiVQABench Vision-language models answer multiple-choice questions that require image evidence and linked Wikipedia and Wikidata knowledge. | IBM Research | General visual questions | University | Not in cards | May 2026 | ||||
| WildGraphBench GraphRAG systems answer 1,100 questions over long, mixed reference documents behind Wikipedia articles in 12 topics, from single facts to section summaries. | USTC and Metastone Technology (DeepResearch Bench team) | Reasoning over long documents | University | Not in cards | Feb 2026 | ||||
| WildKhmerST Models find and read Khmer text in photographs of public places. Annotations mark text lines with polygons and transcriptions. | Cambodia Academy of Digital Technology and La Rochelle University collaborators | Documents, charts and text in images | Everyday | Not in cards | Jun 2025 | ||||
| Winoground Models match two images to two captions that use the same words in a different order. | Tristan Thrush and colleagues | General visual questions | School | Not in cards | Apr 2022 | ||||
| Wisesight Sentiment Models classify Thai social media posts about products and services. SEA-HELM tests positive, neutral, and negative sentiment and excludes the original question label. | PyThaiNLP and Wisesight | Reading in many languages | Everyday | Not in cards | Sep 2019 | ||||
| WixQA Models retrieve Wix support articles and answer customer questions from those articles. The authors score retrieval and answer generation separately. | Dvir Cohen and colleagues | Running a business | Everyday | Not in cards | Apr 2025 | ||||
| WMT 2014 News Translation Models translate news text between English and other languages. The shared task compares submitted translations on a common test set. | Workshop on Statistical Machine Translation | Translation | University | Not in cards | Jun 2014 | ||||
| WMT 2016 News Translation Models translate news between languages. The harness compares translations with human references from the 2016 shared task. | Ondřej Bojar and the WMT 2016 organizers | Translation | University | Not in cards | Aug 2016 | ||||
| WMT24 Icelandic Proper Names Models translate English sentences containing place names or personal names into Icelandic. The scorer checks the Icelandic name and its grammatical form. | Árni Magnússon Institute for Icelandic Studies | Translation | Everyday | Not in cards | Jun 2024 | ||||
| WMT26 Multilingual Instruction Shared Task v2026 Models answer questions from passages, summarize papers across languages and follow writing instructions. The overall ranking averages ranks across the three tasks. | WMT multilingual instruction organizers | Reading in many languages | University | Not in cards | Jan 2026 | ||||
| WMT25 Multilingual Instruction Shared Task v2025 Replaced in Jan 2026 | Reasoning in many languages | Not in cards | Mar 2025 | ||||||
| WMT26 Video Subtitle Translation Models translate timed Chinese subtitles into five target languages using a synopsis for context. Human reviewers check the model judge’s assessments. | WMT video subtitle translation organizers | Translation | University | Not in cards | Apr 2026 | ||||
| WOLF Models play Werewolf to measure deception production and deception detection as separate abilities. | WOLF authors | Multi-agent games | Everyday | Not in cards | Dec 2025 | ||||
| Word-Level BISINDO Models identify 32 signs from the Banten variety of Indonesian Sign Language in video clips. | BINUS University | Video understanding | Everyday | Not in cards | May 2025 | ||||
| WorkArena Web agents do routine knowledge-work tasks in a live ServiceNow instance, such as filling forms and filtering lists. WorkArena++ adds 682 compositional tasks at levels 2 and 3. | ServiceNow Research | Web browsing | University | Not in cards | Mar 2024 | ||||
| WorkIndex Agents complete everyday WorkBuddy tasks. Tencent combines eight tests covering office files, audio and video work, and visual questions. | Tencent | Composite work indices | University | Not in cards | Nov 2026 | ||||
| WorkRB Skill Extraction Challenge 2026 Models rank occupational skills for vacancy sentences and alternative skill names. Organizers average graded ranking scores at depth 100 across five test sets. | TechWolf, Avature, and Ghent University | Other professions | University | Not in cards | Jun 2026 | ||||
| WorldModelBench Video generators judged as world models on instruction following and physics adherence in application domains. Human labels and a fine-tuned judge. | Video world models | Not in cards | Feb 2025 | ||||||
| WorldModelGym World models predict returns for five candidate action sequences across 100+ Atari, Meta-World, DMC and classic-control tracks. Scored by normalized regret. | Reka Labs | Interactive world models | Not in cards | Jul 2026 | |||||
| WorldOlympiad Models generate long videos for gaming, robotics, and everyday scenes. The evaluator checks physical rules, reconstructed geometry, and instruction following across video segments. | Zhejiang University, Alibaba DAMO Academy, Hong Kong University of Science and Technology, and collaborators | Video world models | Everyday | Not in cards | Jun 2026 | ||||
| WorldPrediction Agents predict and plan in procedural environments with long horizons. Tasks test whether models maintain a useful world model while acting. | Meta FAIR and Superintelligence Labs | Interactive world models | University | Not in cards | Jun 2025 | ||||
| WorldScore 3,000 next-scene generation tasks with camera-trajectory layouts. Scores 3D/4D and video world generators on controllability, quality, dynamics. | Stanford | Video world models | Not in cards | Apr 2025 | |||||
| WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. | Video world models | Not in cards | Oct 2024 | ||||||
| Writing Style Matching (apaz) Models continue a story in the author’s style. A judge compares the generated continuation with the reference continuation and assesses whether the passages sound alike. | apaz | Taste and style | Everyday | Not in cards | Nov 2025 | ||||
| WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. | Medicine | Expert | Not in cards | Jan 2026 | |||||
| WSYue-eval Models transcribe short and long Cantonese recordings. The suite also evaluates speech synthesis. | Longhao Li and colleagues | Speech recognition and translation | Everyday | Not in cards | Sep 2025 | ||||
| x-stance Models identify whether a political comment supports or opposes the proposition in a question. | Jannis Vamvas and colleagues | Reading in many languages | School | Not in cards | Mar 2020 | ||||
| XACLE Challenge 2026 Models listen to generated audio and predict how closely it matches an English description. Organizers rank predictions by their correlation with listener scores. | XACLE Challenge organizers | Audio understanding and voice | Everyday | Not in cards | Jul 2025 | ||||
| xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. | xAI (SpaceXAI) | Refusing harmful requests | Not in cards | ||||||
| XL-Sum (English) Models summarize English BBC news articles. ROUGE-1, ROUGE-2, and ROUGE-L compare generated summaries with journalist-written summaries. The current English test split contains 11,535 examples. | Bangladesh University of Engineering and Technology and collaborators | Faithful to sources | Everyday | Not in cards | Jun 2021 | ||||
| XNLIeu Models judge whether a Basque sentence follows from, contradicts, or is neutral toward a premise. The evaluation includes translated and native examples. | Maite Heredia and colleagues | Reading in many languages | School | Not in cards | Apr 2024 | ||||
| XOR-TyDi QA Models retrieve passages across languages to answer questions written by speakers of seven languages. Tasks require English evidence or answers in the question’s language. | Akari Asai and colleagues | Reading in many languages | School | Not in cards | Oct 2020 | ||||
| XSum Models summarize a BBC news article in one sentence. ROUGE-1, ROUGE-2, and ROUGE-L compare the generated summary with the journalist's introductory summary. | University of Edinburgh | Reading comprehension | Everyday | Not in cards | Aug 2018 | ||||
| XTD-200 Models retrieve images and captions across languages using translations of the XTD test set. Results measure how often the matching item ranks first. | Alexander Visheratin | Reading in many languages | School | Not in cards | Sep 2023 | ||||
| XTREME Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages. | Carnegie Mellon University and Google | Reading in many languages | School | Not in cards | Mar 2020 | ||||
| XTREME-R Multilingual models are fine-tuned on English data and tested on 10 tasks in 50 typologically diverse languages, including harder language-agnostic retrieval tasks. It extends XTREME and adds MultiCheckList, a multilingual diagnostic suite. | Google Research and DeepMind with collaborators | Reading in many languages | School | Not in cards | Apr 2021 | ||||
| XWinograd Models resolve ambiguous pronouns in six languages by choosing which noun fits each sentence. | Alexey Tikhonov and colleagues | Reasoning in many languages | Everyday | Not in cards | Jun 2021 | ||||
| Yet Another Kernel Kernel Models optimize GPU kernels for speed while a verifier rejects numerical errors and input-insensitive shortcuts. | HUD AI | GPU kernels | Research frontier | Not in cards | Nov 2026 | ||||
| YorùLect Models transcribe Yorùbá speech and translate dialect text. The speech task covers three varieties. The text task also covers Ìjèbú. | Orevaoghene Ahia and collaborators | Speech recognition and translation | Everyday | Not in cards | Jun 2024 | ||||
| YouCook2 Dense Video Description Models locate cooking steps and describe each step. The score is CIDEr for the complete dense video description task. | University of Michigan | Video captioning | Everyday | Not in cards | Mar 2017 | ||||
| Your Voice, Your Device, Your Language Challenge Models transcribe Kiswahili recordings for offline speech recognition. Zindi ranks submissions by word error rate before assessing the top ten for deployment. | Sartify, ITU AI for Good and Zindi | Speech recognition and translation | Everyday | Not in cards | Jul 2025 | ||||
| YouTube-VIS Models detect, segment, and track object instances in YouTube videos. The SAM 3 report evaluates the 2021 edition. | YouTube-VIS team | Video understanding | Everyday | Not in cards | May 2019 | ||||
| YouTube-VOS Models segment and track objects across frames in YouTube videos. The evaluation distinguishes categories seen during training from previously unseen categories. | Ning Xu and colleagues | Video understanding | Everyday | Not in cards | Sep 2018 | ||||
| Zalo AI Challenge 2022 End-to-End Question Answering Models answer Vietnamese general knowledge questions. HELM scores short answers without supplying a passage. | Zalo AI | Knowledge in many languages | School | Not in cards | |||||
| ZendoWorld Models infer hidden rules in the board game Zendo from labeled examples. | ZendoWorld authors | Puzzles and chess | Everyday | Not in cards | Jul 2026 | ||||
| ZenML Bench Agents build, repair, and investigate machine learning pipelines in ZenML. Eighteen tasks grade the state the framework records. | ZenML | Machine learning engineering | Expert | Not in cards | Sep 2026 | ||||
| Zerolang Editing Agents repair and refactor Zerolang programs through graph editing tools. The evaluation includes 67 held-out synthetic tasks. | pandelis | Fixing real repositories | University | Not in cards | May 2026 | ||||
| Zoomed-in BVCC Models predict how listeners rate synthesized speech when they hear only the highest-rated systems. Listeners rated the same clips again in progressively smaller groups. | Erica Cooper and Junichi Yamagishi | Audio understanding and voice | Everyday | Not in cards | May 2023 |